mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-29 20:14:01 +00:00
[OpenVINO EP] v3.3 Release Documentation update (#9589)
This commit is contained in:
Maajid khan
GitHub
parent
6e7c43767e
commit
da6f161717
2 changed files with 53 additions and 14 deletions
7
docs/build/eps.md
vendored
7
docs/build/eps.md
vendored
|
|
@ -290,16 +290,17 @@ See more information on the OpenVINO Execution Provider [here](../execution-prov
|
|||
| <code>VAD-F_FP32</code> | Intel<sup>®</sup> Vision Accelerator Design with an Intel<sup>®</sup> Arria<sup>®</sup> 10 FPGA |
|
||||
| <code>HETERO:DEVICE_TYPE_1,DEVICE_TYPE_2,DEVICE_TYPE_3...</code> | All Intel<sup>®</sup> silicons mentioned above |
|
||||
| <code>MULTI:DEVICE_TYPE_1,DEVICE_TYPE_2,DEVICE_TYPE_3...</code> | All Intel<sup>®</sup> silicons mentioned above |
|
||||
| <code>AUTO:DEVICE_TYPE_1,DEVICE_TYPE_2,DEVICE_TYPE_3...</code> | All Intel<sup>®</sup> silicons mentioned above |
|
||||
|
||||
Specifying Hardware Target for HETERO or Multi-Device Build:
|
||||
Specifying Hardware Target for HETERO or Multi or AUTO device Build:
|
||||
|
||||
HETERO:DEVICE_TYPE_1,DEVICE_TYPE_2,DEVICE_TYPE_3...
|
||||
The DEVICE_TYPE can be any of these devices from this list ['CPU','GPU','MYRIAD','FPGA','HDDL']
|
||||
|
||||
A minimum of two device's should be specified for a valid HETERO or Multi-Device Build.
|
||||
A minimum of two device's should be specified for a valid HETERO or MULTI or AUTO device build.
|
||||
|
||||
```
|
||||
Example's: HETERO:MYRIAD,CPU or HETERO:HDDL,GPU,CPU or MULTI:MYRIAD,GPU,CPU
|
||||
Example's: HETERO:MYRIAD,CPU or AUTO:GPU,CPU or MULTI:MYRIAD,GPU,CPU
|
||||
```
|
||||
|
||||
#### Disable subgraph partition Feature
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ OpenVINO Execution Provider enables deep learning inference on Intel CPUs, Intel
|
|||
{:toc}
|
||||
|
||||
## Install
|
||||
Pre-built packages and Docker images are published for ONNX Runtime with OpenVINO by Intel for each release.
|
||||
Pre-built packages and Docker images are published for ONNX Runtime with OpenVINO by Intel for each release.
|
||||
* Python wheels: [intel/onnxruntime](https://github.com/intel/onnxruntime/releases)
|
||||
* Docker image: [openvino/onnxruntime_ep_ubuntu18](https://hub.docker.com/r/openvino/onnxruntime_ep_ubuntu18)
|
||||
|
||||
|
|
@ -60,15 +60,25 @@ More consistent performance, since the devices can now share the inference burde
|
|||
For more information on Multi-Device plugin of OpenVINO, please refer to the following
|
||||
[documentation](https://docs.openvinotoolkit.org/latest/openvino_docs_IE_DG_supported_plugins_MULTI.html#introducing_multi_device_execution).
|
||||
|
||||
### Save/Load blob feature for OpenVINO EP
|
||||
### Auto-Device Execution for OpenVINO EP
|
||||
|
||||
Use "AUTO:<device 1><device 2>.." as the device name to delegate selection of an actual accelerator to OpenVINO. With the 2021.4 release, Auto-device internally recognizes and selects devices from CPU, integrated GPU and discrete Intel GPUs (when available) depending on the device capabilities and the characteristic of CNN models, for example, precisions. Then Auto-device assigns inference requests to the selected device.
|
||||
|
||||
From the application point of view, this is just another device that handles all accelerators in full system.
|
||||
|
||||
For more information on Auto-Device plugin of OpenVINO, please refer to the following
|
||||
[documentation](https://docs.openvino.ai/cn/latest/openvino_docs_IE_DG_supported_plugins_AUTO.html).
|
||||
|
||||
### Model caching feature for OpenVINO EP
|
||||
The model caching setting enables blobs with Myriadx(VPU) and as cl_cache files with iGPU.
|
||||
#### Save/Load blob capability for Myriadx(VPU)
|
||||
This feature enables users to save and load the blobs directly. These pre-compiled blobs can be directly loaded on to the specific hardware device target and inferencing can be done. This feature is only supported on MyriadX(VPU) hardware device target.
|
||||
|
||||
#### CL Cache capability for iGPU
|
||||
|
||||
Starting from OpenVINO 2021.4 version, this feature is supported in OpenVINO-EP using Model caching mechanism from OpenVINO.[documentation](https://docs.openvinotoolkit.org/latest/openvino_docs_IE_DG_Model_caching_overview.html).
|
||||
|
||||
This feature is now available for MyriadX(VPU) and iGPU from OpenVINO 2021.4. Currently, this feature is not supported for Intel CPU's.
|
||||
|
||||
Improved overall inferencing time, since this feature eliminates the preliminary steps of creating a network from the model. Here, the pre-compiled blob is directly imported on to the device target.
|
||||
This feature enables users to save and load the cl_cache files directly. These cl_cache files can be directly loaded on to igpu hardware device target and inferencing can be done. This feature is only supported on iGPU hardware device target.
|
||||
|
||||
There are two different methods of exercising this feature:
|
||||
|
||||
|
|
@ -76,7 +86,7 @@ There are two different methods of exercising this feature:
|
|||
|
||||
This flow can be enabled by setting the runtime config option 'use_compiled_network' to True while using the c++/python API'S. This config option acts like a switch to on and off the feature.
|
||||
|
||||
The blobs are saved and loaded from a directory named 'ov_compiled_blobs' from the executable path by default. This path however can be overridden using another runtime config option 'blob_dump_path' which is used to explicitly specify the path where you would like to dump and load the blobs from when already using the use_compiled_network(save/load blob) setting.
|
||||
The blobs are saved and loaded from a directory named 'ov_compiled_blobs' from the executable path by default. This path however can be overridden using another runtime config option 'blob_dump_path' which is used to explicitly specify the path where you would like to dump and load the blobs (VPU) or cl_cache(iGPU) files from when already using the use_compiled_network(model caching) setting.
|
||||
|
||||
Refer to [Configuration Options](#configuration-options) for more information about using these runtime options.
|
||||
|
||||
|
|
@ -130,7 +140,33 @@ Note:
|
|||
|
||||
3. Install the latest ONNX Python package using pip to run these ONNX Python API's successfully.
|
||||
|
||||
## Configuration Options
|
||||
### Support for IO Buffer Optimization
|
||||
|
||||
To enable IO Buffer Optimization we have to set OPENCL_LIBS, OPENCL_INCS environment variables before build. For IO Buffer Optimization, the model must be fully supported on
|
||||
OpenVINO and we must provide in the remote context cl_context void pointer as C++ Configuration Option. We can provide cl::Buffer address as Input using GPU Memory Allocator for input and output.
|
||||
|
||||
Example:
|
||||
```bash
|
||||
//Set up a remote context
|
||||
cl::Context _context;
|
||||
.....
|
||||
// Set the context through openvino options
|
||||
OrtOpenVINOProviderOptions options;
|
||||
options.context = (void *) _context.get() ;
|
||||
.....
|
||||
//Define the Memory area
|
||||
Ort::MemoryInfo info_gpu("IGPU", OrtAllocatorType::OrtDeviceAllocator, 0, OrtMemTypeDefault);
|
||||
//Create a shared buffer , fill in with data
|
||||
cl::Buffer shared_buffer(_context, CL_MEM_READ_WRITE, imgSize, NULL, &err);
|
||||
....
|
||||
//Cast it to void*, and wrap it as device pointer for Ort::Value
|
||||
void *shared_buffer_void = static_cast<void *>(&shared_buffer);
|
||||
Ort::Value inputTensors = Ort::Value::CreateTensor(
|
||||
info_gpu, shared_buffer_void, imgSize, inputDims.data(),
|
||||
inputDims.size(), ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT);
|
||||
```
|
||||
|
||||
### Configuration Options
|
||||
|
||||
OpenVINO EP can be configured with certain options at runtime that control the behavior of the EP. These options can be set as key-value pairs as below:-
|
||||
|
||||
|
|
@ -153,6 +189,7 @@ options.device_id = "";
|
|||
options.num_of_threads = 8;
|
||||
options.use_compiled_network = false;
|
||||
options.blob_dump_path = "";
|
||||
options.context = 0x123456ff;
|
||||
SessionOptionsAppendExecutionProvider_OpenVINO(session_options, &options);
|
||||
```
|
||||
|
||||
|
|
@ -161,21 +198,22 @@ The following table lists all the available configuration options and the Key-Va
|
|||
|
||||
| **Key** | **Key type** | **Allowable Values** | **Value type** | **Description** |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| device_type | string | CPU_FP32, GPU_FP32, GPU_FP16, MYRIAD_FP16, VAD-M_FP16, VAD-F_FP32, Any valid Hetero combination, Any valid Multi-Device combination | string | Overrides the accelerator hardware type and precision with these values at runtime. If this option is not explicitly set, default hardware and precision specified during build time is used. |Overrides the accelerator hardware type and precision with these values at runtime. If this option is not explicitly set, default hardware and precision specified during build time is used. |
|
||||
| device_type | string | CPU_FP32, GPU_FP32, GPU_FP16, MYRIAD_FP16, VAD-M_FP16, VAD-F_FP32, Any valid Hetero combination, Any valid Multi or Auto devices combination | string | Overrides the accelerator hardware type and precision with these values at runtime. If this option is not explicitly set, default hardware and precision specified during build time is used. |Overrides the accelerator hardware type and precision with these values at runtime. If this option is not explicitly set, default hardware and precision specified during build time is used. |
|
||||
| device_id | string | Any valid OpenVINO device ID | string | Selects a particular hardware device for inference. The list of valid OpenVINO device ID's available on a platform can be obtained either by Python API (`onnxruntime.capi._pybind_state.get_available_openvino_device_ids()`) or by [OpenVINO C/C++ API](https://docs.openvinotoolkit.org/latest/classInferenceEngine_1_1Core.html#acb212aa879e1234f51b845d2befae41c). If this option is not explicitly set, an arbitrary free device will be automatically selected by OpenVINO runtime.|
|
||||
| enable_vpu_fast_compile | string | True/False | boolean | This option is only available for MYRIAD_FP16 VPU devices. During initialization of the VPU device with compiled model, Fast-compile may be optionally enabled to speeds up the model's compilation to VPU device specific format. This in-turn speeds up model initialization time. However, enabling this option may slowdown inference due to some of the optimizations not being fully applied, so caution is to be exercised while enabling this option. |
|
||||
| num_of_threads | string | Any unsigned positive number other than 0 | size_t | Overrides the accelerator default value of number of threads with this value at runtime. If this option is not explicitly set, default value of 8 is used during build time. |
|
||||
| use_compiled_network | string | True/False | boolean | This option is only available for MYRIAD_FP16 VPU devices for both Linux and Windows and it enables save/load blob functionality. It can be used to directly import pre-compiled blobs if exists or dump a pre-compiled blob at the executable path. |
|
||||
| blob_dump_path | string | Any valid string path on the hardware target | string | Explicitly specify the path where you would like to dump and load the blobs for the save/load blob feature when use_compiled_network setting is enabled . This overrides the default path.|
|
||||
| context | string | OpenCL Context | void* | This option is only alvailable when OpenVINO EP is built with OpenCL flags enabled. It takes in the remote context i.e the cl_context address as a void pointer.|
|
||||
|
||||
Valid Hetero or Multi-Device combinations:
|
||||
Valid Hetero or Multi or Auto Device combinations:
|
||||
HETERO:<DEVICE_TYPE_1>,<DEVICE_TYPE_2>,<DEVICE_TYPE_3>...
|
||||
The <DEVICE_TYPE> can be any of these devices from this list ['CPU','GPU','MYRIAD','FPGA','HDDL']
|
||||
|
||||
A minimum of two DEVICE_TYPE'S should be specified for a valid HETERO or Multi-Device Build.
|
||||
|
||||
Example:
|
||||
HETERO:MYRIAD,CPU HETERO:HDDL,GPU,CPU MULTI:MYRIAD,GPU,CPU
|
||||
HETERO:MYRIAD,CPU AUTO:GPU,CPU MULTI:MYRIAD,GPU,CPU
|
||||
|
||||
### Other configuration settings
|
||||
#### Onnxruntime Graph Optimization level
|
||||
|
|
@ -402,7 +440,7 @@ In order to showcase what you can do with the OpenVINO Execution Provider for ON
|
|||
[Object detection with YOLOv4 in Python](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/python/OpenVINO_EP/yolov4_object_detection)
|
||||
|
||||
### C/C++ API
|
||||
[Image classification with Squeezenet in CPP](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_cxx/OpenVINO_EP/squeezenet_classification)
|
||||
[Image classification with Squeezenet in CPP](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_cxx/OpenVINO_EP)
|
||||
|
||||
### Csharp API
|
||||
[Object detection with YOLOv3 in C#](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_sharp/OpenVINO_EP/yolov3_object_detection)
|
||||
|
|
|
|||
Loading…
Reference in a new issue