saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-29 20:14:01 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Ort openvino npu 1.18.0 doc (#20707)

Browse source 
### Description
-These changes reflect the latest changes we have made to enable NPU
-Export OpenVINO Blob to ONNX Operator
-Enable QDQ Optimizations Passes
- Separate Device/Precision



### Motivation and Context
These changes reflect the latest changes we have made to enable NPU. 
These changes reflect the latest changes we have made to enable rel-1.18
- If it fixes an open issue, please link to the issue here. -->

---------

Co-authored-by: Preetha Veeramalai <preetha.veeramalai@intel.com>
This commit is contained in:
sfatimar 2024-05-20 11:40:24 +05:30 committed by GitHub
parent 5fd617acc0
commit 3909f23ee6
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 38 additions and 30 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

25
docs/build/eps.md vendored
View file

@ -250,14 +250,13 @@ See more information on the OpenVINO™ Execution Provider [here](../execution-p
### Prerequisites
{: .no_toc }
1. Install the OpenVINO™ offline/online installer from Intel<sup>®</sup> Distribution of OpenVINO™<sup>TM</sup> Toolkit **Release 2023.1** for the appropriate OS and target hardware:
* [Windows - CPU, GPU](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/download.html?VERSION=v_2023_1_0&OP_SYSTEM=WINDOWS&DISTRIBUTION=ARCHIVE).
1. Install the OpenVINO™ offline/online installer from Intel<sup>®</sup> Distribution of OpenVINO™<sup>TM</sup> Toolkit **Release 2024.1** for the appropriate OS and target hardware:
* [Windows - CPU, GPU, NPU](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/download.html?VERSION=v_2023_1_0&OP_SYSTEM=WINDOWS&DISTRIBUTION=ARCHIVE).
* [Linux - CPU, GPU](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/download.html?VERSION=v_2023_1_0&OP_SYSTEM=LINUX&DISTRIBUTION=ARCHIVE)
Follow [documentation](https://docs.openvino.ai/latest/index.html) for detailed instructions.
Follow [documentation](https://docs.openvino.ai/2024/home.html) for detailed instructions.
*2023.1 is the recommended OpenVINO™ version. [OpenVINO™ 2022.1](https://docs.openvino.ai/archive/2022.1/index.html) is minimal OpenVINO™ version requirement.*
*The minimum ubuntu version to support 2023.1 is 18.04.*
*2024.1 is the current recommended OpenVINO™ version. [OpenVINO™ 2023.1](https://docs.openvino.ai/archive/2023.1/home.html) is minimal OpenVINO™ version requirement.*
2. Configure the target hardware with specific follow on instructions:
* To configure Intel<sup>®</sup> Processor Graphics(GPU) please follow these instructions: [Windows](https://docs.openvino.ai/latest/openvino_docs_install_guides_configurations_for_intel_gpu.html#gpu-guide-windows), [Linux](https://docs.openvino.ai/latest/openvino_docs_install_guides_configurations_for_intel_gpu.html#linux)
@ -291,7 +290,7 @@ See more information on the OpenVINO™ Execution Provider [here](../execution-p
./build.sh --config RelWithDebInfo --use_openvino <hardware_option> --build_shared_lib --build_wheel
```
* `--build_wheel` Creates python wheel file in dist/ folder. Enable it when building from source and/or while building with CXX11_ABI=1 of OpenVINO.
* `--build_wheel` Creates python wheel file in dist/ folder. Enable it when building from source.
* `--use_openvino` builds the OpenVINO™ Execution Provider in ONNX Runtime.
* `<hardware_option>`: Specifies the default hardware target for building OpenVINO™ Execution Provider. This can be overriden dynamically at runtime with another option (refer to [OpenVINO™-ExecutionProvider](../execution-providers/OpenVINO-ExecutionProvider.md#summary-of-options) for more details on dynamic device selection). Below are the options for different Intel target devices.
@ -299,13 +298,11 @@ Refer to [Intel GPU device naming convention](https://docs.openvino.ai/latest/op
| Hardware Option | Target Device |
| --------------- | ------------------------|
| <code>CPU_FP32</code> | Intel<sup>®</sup> CPUs |
| <code>GPU_FP32</code> | Intel<sup>®</sup> Integrated Graphics |
| <code>GPU_FP16</code> | Intel<sup>®</sup> Integrated Graphics with FP16 quantization of models |
| <code>GPU.0_FP32</code> | Intel<sup>®</sup> Integrated Graphics |
| <code>GPU.0_FP16</code> | Intel<sup>®</sup> Integrated Graphics with FP16 quantization of models |
| <code>GPU.1_FP32</code> | Intel<sup>®</sup> Discrete Graphics |
| <code>GPU.1_FP16</code> | Intel<sup>®</sup> Discrete Graphics with FP16 quantization of models |
| <code>CPU</code> | Intel<sup>®</sup> CPUs |
| <code>GPU</code> | Intel<sup>®</sup> Integrated Graphics |
| <code>GPU.0</code> | Intel<sup>®</sup> Integrated Graphics |
| <code>GPU.1</code> | Intel<sup>®</sup> Discrete Graphics |
| <code>NPU</code> | Intel<sup>®</sup> Neural Processor Unit |
| <code>HETERO:DEVICE_TYPE_1,DEVICE_TYPE_2,DEVICE_TYPE_3...</code> | All Intel<sup>®</sup> silicons mentioned above |
| <code>MULTI:DEVICE_TYPE_1,DEVICE_TYPE_2,DEVICE_TYPE_3...</code> | All Intel<sup>®</sup> silicons mentioned above |
| <code>AUTO:DEVICE_TYPE_1,DEVICE_TYPE_2,DEVICE_TYPE_3...</code> | All Intel<sup>®</sup> silicons mentioned above |
@ -313,7 +310,7 @@ Refer to [Intel GPU device naming convention](https://docs.openvino.ai/latest/op
Specifying Hardware Target for HETERO or Multi or AUTO device Build:
HETERO:DEVICE_TYPE_1,DEVICE_TYPE_2,DEVICE_TYPE_3...
The DEVICE_TYPE can be any of these devices from this list ['CPU','GPU']
The DEVICE_TYPE can be any of these devices from this list ['CPU','GPU', 'NPU']
A minimum of two device's should be specified for a valid HETERO or MULTI or AUTO device build.

43
docs/execution-providers/OpenVINO-ExecutionProvider.md
View file

@ -9,7 +9,7 @@ redirect_from: /docs/reference/execution-providers/OpenVINO-ExecutionProvider
# OpenVINO™ Execution Provider
{: .no_toc }
Accelerate ONNX models on Intel CPUs, GPUs with Intel OpenVINO™ Execution Provider. Please refer to [this](https://software.intel.com/en-us/openvino-toolkit/hardware) page for details on the Intel hardware supported.
Accelerate ONNX models on Intel CPUs, GPUs, NPU with Intel OpenVINO™ Execution Provider. Please refer to [this](https://software.intel.com/en-us/openvino-toolkit/hardware) page for details on the Intel hardware supported.
## Contents
{: .no_toc }
@ -31,7 +31,6 @@ ONNX Runtime OpenVINO™ Execution Provider is compatible with three lastest rel
|ONNX Runtime|OpenVINO™|Notes|
|---|---|---|
|1.17.1|2023.3|[Details](https://github.com/intel/onnxruntime/releases/tag/v5.2)|
|1.17.1|2023.2|[Details](https://github.com/intel/onnxruntime/releases/tag/v5.2)|
|1.16.0|2023.1|[Details](https://github.com/intel/onnxruntime/releases/tag/v5.1)|
|1.15.0|2023.0|[Details](https://github.com/intel/onnxruntime/releases/tag/v5.0.0)|
|1.14.0|2022.3|[Details](https://github.com/intel/onnxruntime/releases/tag/v4.3)|
@ -98,7 +97,7 @@ Enables [OpenCL queue throttling](https://docs.openvino.ai/latest/groupov_runtim
OpenVINO™ supports [model caching](https://docs.openvino.ai/2024/openvino-workflow/running-inference/optimize-inference/optimizing-latency/model-caching-overview.html).
From OpenVINO™ 2023.1 version, model caching feature is supported on CPU, GPU along with kernel caching on iGPU, dGPU.
Model caching feature is supported on CPU, NPU, GPU along with kernel caching on iGPU, dGPU.
This feature enables users to save and load the blob file directly on to the hardware device target and perform inference with improved Inference Latency.
@ -108,13 +107,13 @@ This feature also allows user to save kernel caching as cl_cache files for model
#### <b> Enabling Model Caching via Runtime options using c++/python API's.</b>
This flow can be enabled by setting the runtime config option 'cache_dir' specifying the path to dump and load the blobs (CPU, iGPU, dGPU) or cl_cache(iGPU, dGPU) while using the c++/python API'S.
This flow can be enabled by setting the runtime config option 'cache_dir' specifying the path to dump and load the blobs (CPU, NPU, iGPU, dGPU) or cl_cache(iGPU, dGPU) while using the c++/python API'S.
Refer to [Configuration Options](#configuration-options) for more information about using these runtime options.
### Support for INT8 Quantized models
Int8 models are supported on CPU and GPU.
Int8 models are supported on CPU, GPU and NPU.
### Support for Weights saved in external files
@ -174,7 +173,7 @@ OpenVINO™ Execution Provider for ONNX Runtime allows multiple stream execution
### Auto-Device Execution for OpenVINO EP
Use `AUTO:<device 1><device 2>..` as the device name to delegate selection of an actual accelerator to OpenVINO™. Auto-device internally recognizes and selects devices from CPU, integrated GPU and discrete Intel GPUs (when available) depending on the device capabilities and the characteristic of CNN models, for example, precisions. Then Auto-device assigns inference requests to the selected device.
Use `AUTO:<device 1><device 2>..` as the device name to delegate selection of an actual accelerator to OpenVINO™. Auto-device internally recognizes and selects devices from CPU, integrated GPU, discrete Intel GPUs (when available) and NPU (when available) depending on the device capabilities and the characteristic of CNN models, for example, precisions. Then Auto-device assigns inference requests to the selected device.
From the application point of view, this is just another device that handles all accelerators in full system.
@ -200,7 +199,14 @@ Multi-Device plugin automatically assigns inference requests to available comput
For more information on Multi-Device plugin of OpenVINO™, please refer to the
[Intel OpenVINO™ Multi Device Plugin](https://docs.openvino.ai/latest/openvino_docs_OV_UG_Running_on_multiple_devices.html).
### Export OpenVINO Compiled Blob
Export the OpenVINO compiled blob as an ONNX model. Using this ONNX model for subsequent inferences avoids model recompilation and could have a positive impact on Session creation time. The exported model is saved to the same directory as the source model with the suffix -ov_{device}_blob.onnx where device can be one of the supported like CPU or NPU. This feature is currently enabled for fully supported models only.
Refer to [Configuration Options](#configuration-options) for more information about using these runtime options.
### Enable QDQ Optimizations Passes
Optimizes ORT quantized models for the NPU device to only keep QDQs for supported ops and optimize for performance and accuracy.Generally this feature will give better performance/accuracy with ORT Optimizations disabled.
Refer to [Configuration Options](#configuration-options) for more information about using these runtime options.
## Configuration Options
OpenVINO™ Execution Provider can be configured with certain options at runtime that control the behavior of the EP. These options can be set as key-value pairs as below:-
@ -218,8 +224,8 @@ The session configuration options are passed to SessionOptionsAppendExecutionPro
```
std::unordered_map<std::string, std::string> options;
options[device_type] = "GPU_FP32";
options[device_id] = "";
options["device_type"] = "GPU";
options["precision"] = "FP32";
options[num_of_threads] = "8";
options[num_streams] = "8";
options[cache_dir] = "";
@ -233,8 +239,8 @@ The session configuration options are passed to SessionOptionsAppendExecutionPro
```
OrtOpenVINOProviderOptions options;
options.device_type = "GPU_FP32";
options.device_id = "";
options.device_type = "GPU";
options.precision = "FP32";
options.num_of_threads = 8;
options.cache_dir = "";
options.context = 0x123456ff;
@ -263,31 +269,35 @@ The following table lists all the available configuration options for API 2.0 an
| **Key** | **Key type** | **Allowable Values** | **Value type** | **Description** |
| --- | --- | --- | --- | --- |
| device_type | string | CPU_FP32, CPU_FP16, GPU_FP32, GPU_FP16, GPU.0_FP32, GPU.1_FP32, GPU.0_FP16, GPU.1_FP16 based on the avaialable GPUs, Any valid Hetero combination, Any valid Multi or Auto devices combination | string | Overrides the accelerator hardware type and precision with these values at runtime. If this option is not explicitly set, default hardware and precision specified during build time is used. |Overrides the accelerator hardware type and precision with these values at runtime. If this option is not explicitly set, default hardware and precision specified during build time is used. |
| device_id | string | Any valid OpenVINO device ID | string | Selects a particular hardware device for inference. The list of valid OpenVINO device ID's available on a platform can be obtained either by Python API (`onnxruntime.capi._pybind_state.get_available_openvino_device_ids()`) or by [OpenVINO C/C++ API](https://docs.openvino.ai/latest/classInferenceEngine_1_1Core.html). If this option is not explicitly set, an arbitrary free device will be automatically selected by OpenVINO runtime.|
| device_type | string | CPU, NPU, GPU, GPU.0, GPU.1 based on the avaialable GPUs, NPU, Any valid Hetero combination, Any valid Multi or Auto devices combination | string | Overrides the accelerator hardware type with these values at runtime. If this option is not explicitly set, default hardware specified during build is used. |
| precision | string | FP32, FP16, ACCURACY based on the device_type chosen | string | Supported precisions for HW {CPU:FP32, GPU:[FP32, FP16, ACCURACY], NPU:FP16}. Default precision for HW for optimized performance {CPU:FP32, GPU:FP16, NPU:FP16}. To execute model with the default input precision, select ACCURACY precision type. |
| num_of_threads | string | Any unsigned positive number other than 0 | size_t | Overrides the accelerator default value of number of threads with this value at runtime. If this option is not explicitly set, default value of 8 during build time will be used for inference. |
| num_streams | string | Any unsigned positive number other than 0 | size_t | Overrides the accelerator default streams with this value at runtime. If this option is not explicitly set, default value of 1, performance for latency is used during build time will be used for inference. |
| cache_dir | string | Any valid string path on the hardware target | string | Explicitly specify the path to save and load the blobs enabling model caching feature.|
| context | string | OpenCL Context | void* | This option is only available when OpenVINO EP is built with OpenCL flags enabled. It takes in the remote context i.e the cl_context address as a void pointer.|
| enable_opencl_throttling | string | True/False | boolean | This option enables OpenCL queue throttling for GPU devices (reduces CPU utilization when using GPU). |
| enable_qdq_optimizer | string | True/False | boolean | This option enables QDQ Optimization to improve model performance and accuracy on NPU. |
| export_ep_ctx_blob | string | True/False | boolean | This options enables exporting the OpenVINO Compiled Blob as an ONNX Operator EPContext. |
Valid Hetero or Multi or Auto Device combinations:
HETERO:<DEVICE_TYPE_1>,<DEVICE_TYPE_2>,<DEVICE_TYPE_3>...
The <DEVICE_TYPE> can be any of these devices from this list ['CPU','GPU']
The <DEVICE_TYPE> can be any of these devices from this list ['CPU','GPU', 'NPU']
A minimum of two DEVICE_TYPE'S should be specified for a valid HETERO or Multi-Device Build.
Example:
HETERO:GPU,CPU AUTO:GPU,CPU MULTI:GPU,CPU
Deprecated device_type option :
CPU_FP32, GPU_FP32, GPU_FP16 as still supported. It will be deprectaed in the future release. Kindly upgrade to latest device_type and precision option.
## Support Coverage
**ONNX Layers supported using OpenVINO**
The table below shows the ONNX layers supported and validated using OpenVINO™ Execution Provider.The below table also lists the Intel hardware support for each of the layers. CPU refers to Intel<sup>®</sup>
Atom, Core, and Xeon processors. GPU refers to the Intel Integrated Graphics. Intel Discrete Graphics
Atom, Core, and Xeon processors. GPU refers to the Intel Integrated Graphics. Intel Discrete Graphics. For NPU if an op is not supported we fallback to CPU.
| **ONNX Layers** | **CPU** | **GPU** |
| --- | --- | --- |
@ -338,7 +348,7 @@ Atom, Core, and Xeon processors. GPU refers to the Intel Integrated Graphics. In
| GlobalAveragePool | Yes | Yes |
| GlobalLpPool | Yes | Yes |
| GlobalMaxPool | Yes | Yes |
| Greater | Yes | Yes | Yes |
| Greater | Yes | Yes |
| GreaterOrEqual | Yes | Yes |
| GridSample | Yes | No |
| HardMax | Yes | Yes |
@ -430,7 +440,8 @@ Atom, Core, and Xeon processors. GPU refers to the Intel Integrated Graphics. In
### Topology Support
Below topologies from ONNX open model zoo are fully supported on OpenVINO™ Execution Provider and many more are supported through sub-graph partitioning
Below topologies from ONNX open model zoo are fully supported on OpenVINO™ Execution Provider and many more are supported through sub-graph partitioning.
For NPU is model is not supported we fallback to CPU.
### Image Classification Networks