saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-30 20:18:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Ov 2022.3 4.3 doc (#14252)

Browse source
This commit is contained in:
sfatimar 2023-01-20 05:27:23 +05:30 committed by GitHub
parent 4a9e20aa30
commit f4d3d5f975
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 224 additions and 260 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

484
docs/execution-providers/OpenVINO-ExecutionProvider.md
View file

@ -20,9 +20,10 @@ Accelerate ONNX models on Intel CPUs, GPUs and VPUs with Intel OpenVINO™ Execu
## Install
Pre-built packages and Docker images are published for OpenVINO™ Execution Provider for ONNX Runtime by Intel for each release.
* OpenVINO™ Execution Provider for ONNX Runtime Release page: [Latest v4.3 Release](https://github.com/intel/onnxruntime/releases)
* OpenVINO™ Execution Provider for ONNX Runtime Release page: [Latest v4.2 Release](https://github.com/intel/onnxruntime/releases)
* Python wheels Ubuntu/Windows: [onnxruntime-openvino](https://pypi.org/project/onnxruntime-openvino/)
* Docker image: [openvino/onnxruntime_ep_ubuntu18](https://hub.docker.com/r/openvino/onnxruntime_ep_ubuntu18)
* Docker image: [openvino/onnxruntime_ep_ubuntu20](https://hub.docker.com/r/openvino/onnxruntime_ep_ubuntu20)
## Requirements
@ -30,6 +31,7 @@ ONNX Runtime OpenVINO™ Execution Provider is compatible with three lastest rel
|ONNX Runtime|OpenVINO™|Notes|
|---|---|---|
|1.13.0|2022.3|[Details](https://github.com/intel/onnxruntime/releases/tag/v4.3)|
|1.13.0|2022.2|[Details](https://github.com/intel/onnxruntime/releases/tag/v4.2)|
|1.11.0|2022.1|[Details](https://github.com/intel/onnxruntime/releases/tag/v4.0)|
|1.10.0|2021.4.2|[Details](https://github.com/intel/onnxruntime/releases/tag/v3.4)|
@ -41,13 +43,6 @@ ONNX Runtime OpenVINO™ Execution Provider is compatible with three lastest rel
For build instructions, please see the [BUILD page](../build/eps.md#openvino).
**Announcement: OpenVINO™ Execution Provider for ONNX Runtime v4.0 Release with OpenVINO™ 2022.1 version**:
* The new OpenVINO™ Execution Provider for ONNX Runtime v4.0 Release introduces a new version of OpenVINO™ API (API 2.0). For more information on the changes and transition steps, see the [transition guide](https://docs.openvino.ai/latest/openvino_2_0_transition_guide.html)
* With this release, we have officially stopped supporting CentOS. We will start supporting RHEL OS from
OpenVINO™ 2022.1.
## Usage
**OpenVINO™ Environment**
@ -61,7 +56,7 @@ pip install onnxruntime-openvino==1.13.0
To enable OpenVINO™ Execution Provider with ONNX Runtime on Windows we must install OpenVINO™ separately:
```
pip install openvino==2022.2
pip install openvino==2022.3
```
Code to be added in ONNX Runtime Windows Samples:
```
@ -83,19 +78,11 @@ C:\ <openvino_install_directory>\setupvars.bat
For Running C++/C# ORT Samples with the OpenVINO™ Execution Provider it is must to set up the OpenVINO™ Environment Variables using the full installer package of OpenVINO™.
Initialize the OpenVINO™ environment by running the setupvars script as shown below. This is a required step:
* For Linux run till OpenVINO™ 2021.4 version:
```
$ source <openvino_install_directory>/bin/setupvars.sh
```
* For Linux run from OpenVINO™ 2022.1 version:
* For Linux run:
```
$ source <openvino_install_directory>/setupvars.sh
```
* For Windows run till OpenVINO™ 2021.4 version:
```
C:\ <openvino_install_directory>\bin\setupvars.bat
```
* For Windows run from OpenVINO™ 2022.1 version:
* For Windows run:
```
C:\ <openvino_install_directory>\setupvars.bat
```
@ -131,7 +118,7 @@ For more information on Multi-Device plugin of OpenVINO™, please refer to the
### Auto-Device Execution for OpenVINO EP
Use `AUTO:<device 1><device 2>..` as the device name to delegate selection of an actual accelerator to OpenVINO™. With the 2021.4 release, Auto-device internally recognizes and selects devices from CPU, integrated GPU and discrete Intel GPUs (when available) depending on the device capabilities and the characteristic of CNN models, for example, precisions. Then Auto-device assigns inference requests to the selected device.
Use `AUTO:<device 1><device 2>..` as the device name to delegate selection of an actual accelerator to OpenVINO™. Auto-device internally recognizes and selects devices from CPU, integrated GPU and discrete Intel GPUs (when available) depending on the device capabilities and the characteristic of CNN models, for example, precisions. Then Auto-device assigns inference requests to the selected device.
From the application point of view, this is just another device that handles all accelerators in full system.
@ -140,7 +127,7 @@ For more information on Auto-Device plugin of OpenVINO™, please refer to the
### Model caching feature for OpenVINO™ Execution Provider
The model caching setting enables blobs with Myriadx(VPU) and as cl_cache files with iGPU.
The model caching setting enables blobs with CPU and iGPU.
### OpenCL queue throttling for GPU devices
@ -148,59 +135,33 @@ Enables [OpenCL queue throttling](https://docs.openvino.ai/latest/groupov_runtim
### Model caching
Starting from version 2021.4, OpenVINO™ supports [model caching](https://docs.openvino.ai/latest/openvino_docs_OV_UG_Model_caching_overview.html). With OpenVINO™ 2021.4, it is supported on Myriadx(VPU)
and iGPU.
OpenVINO™ supports [model caching](https://docs.openvino.ai/latest/openvino_docs_OV_UG_Model_caching_overview.html).
From OpenVINO™ 2022.1 version, this feature is supported on Myriadx(VPU), iGPU and CPU.
From OpenVINO™ 2022.1 version, model caching feature is supported on Myriadx(VPU), CPU and kernel caching on iGPU.
iGPU:
This feature enables users to save and load the cl_cache files directly. These cl_cache files can be loaded directly onto the iGPU hardware device target and inferencing can be performed.
From OpenVINO™ 2022.3 version, the model caching feature is also supported on iGPU as preview.
Myriadx(VPU) and CPU:
This feature enables users to save and load the blob file directly. This file can be loaded directly on to the hardware device target and inferencing can be performed.
There are two different methods of exercising this feature:
Kernel Caching on iGPU :
#### <b>option 1. Enabling via Runtime options using c++/python API's.</b>
This feature also allows user to save kernel caching as cl_cache files for models with dynamic input shapes. These cl_cache files can be loaded directly onto the iGPU hardware device target and inferencing can be performed.
#### <b> Enabling Model Caching via Runtime options using c++/python API's.</b>
This flow can be enabled by setting the runtime config option 'use_compiled_network' to True while using the c++/python API'S. This config option acts like a switch to on and off the feature.
The blobs are saved and loaded from a directory named 'ov_compiled_blobs' relative to the executable path by default. This path however can be overridden using the runtime config option 'blob_dump_path' which is used to explicitly specify the path where you would like to dump and load the blobs (VPU, CPU) or cl_cache(iGPU) files from when already using the use_compiled_network(model caching) setting.
The blobs are saved and loaded from a directory named 'ov_compiled_blobs' relative to the executable path by default. This path however can be overridden using the runtime config option 'blob_dump_path' which is used to explicitly specify the path where you would like to dump and load the blobs (CPU, iGPU) or cl_cache(iGPU) files from when already using the use_compiled_network(model caching) setting.
Refer to [Configuration Options](#configuration-options) for more information about using these runtime options.
#### <b>option 2. Importing the pre-compiled blobs directly from the path set by the user.</b>
This flow enables users to import/load the pre-compiled blob directly if available readily. This option is enabled by explicitly setting the path to the blob using environment variables and setting the OV_USE_COMPILED_NETWORK flag to true.
This flow only works for MyriadX(VPU) device.
For Linux:
```
export OV_USE_COMPILED_NETWORK=1
export OV_BLOB_PATH=<path to the blob>
Example: export OV_BLOB_PATH=/home/blobs_dir/model.blob
```
For Windows:
```
set OV_USE_COMPILED_NETWORK=1
set OV_BLOB_PATH=<path to the blob>
Example: set OV_BLOB_PATH=\home\blobs_dir\model.blob
```
compile_tool:
The device specific Myriadx blobs can be generated using an offline tool called compile_tool from OpenVINO™ Toolkit.[documentation](https://docs.openvino.ai/latest/openvino_inference_engine_tools_compile_tool_README.html).
### Support for INT8 Quantized models
Starting from the OpenVINO™ Execution Provider 2021.4 Release, int8 models will be supported on CPU and GPU.
However, int8 support won't be available for VPU.
Int8 models are supported on CPU and GPU.
### Support for Weights saved in external files
Starting from the OpenVINO™ Execution Provider 2021.4 Release, support for external weights is added. OpenVINO™ Execution Provider now supports ONNX models that store weights in external files. It is especially useful for models larger than 2GB because of protobuf limitations.
OpenVINO™ Execution Provider now supports ONNX models that store weights in external files. It is especially useful for models larger than 2GB because of protobuf limitations.
See the [OpenVINO™ ONNX Support documentation](https://docs.openvino.ai/latest/classov_1_1Core.html).
@ -221,7 +182,7 @@ Note:
2. Now, you can use this 'saved_model.onnx' file to infer using your sample. But remember, the weights file location can't be changed. The weights have to be present at /data/weights_data
3. Install the latest ONNX Python package using pip to run these ONNX Python API's successfully.
.
### Support for IO Buffer Optimization
To enable IO Buffer Optimization we have to set OPENCL_LIBS, OPENCL_INCS environment variables before build. For IO Buffer Optimization, the model must be fully supported on OpenVINO™ and we must provide in the remote context cl_context void pointer as C++ Configuration Option. We can provide cl::Buffer address as Input using GPU Memory Allocator for input and output.
@ -281,7 +242,7 @@ The following table lists all the available configuration options and the Key-Va
| **Key** | **Key type** | **Allowable Values** | **Value type** | **Description** |
| --- | --- | --- | --- | --- |
| device_type | string | CPU_FP32, CPU_FP16, GPU_FP32, GPU_FP16, MYRIAD_FP16, VAD-M_FP16, VAD-F_FP32, GPU.0_FP16, GPU.1_FP16, GPU.0_FP16, GPU.0_FP32 based on the avaialable GPUs, Any valid Hetero combination, Any valid Multi or Auto devices combination | string | Overrides the accelerator hardware type and precision with these values at runtime. If this option is not explicitly set, default hardware and precision specified during build time is used. |Overrides the accelerator hardware type and precision with these values at runtime. If this option is not explicitly set, default hardware and precision specified during build time is used. |
| device_type | string | CPU_FP32, CPU_FP16, GPU_FP32, GPU_FP16, MYRIAD_FP16, GPU.0_FP16, GPU.1_FP16, GPU.0_FP16, GPU.0_FP32 based on the avaialable GPUs, Any valid Hetero combination, Any valid Multi or Auto devices combination | string | Overrides the accelerator hardware type and precision with these values at runtime. If this option is not explicitly set, default hardware and precision specified during build time is used. |Overrides the accelerator hardware type and precision with these values at runtime. If this option is not explicitly set, default hardware and precision specified during build time is used. |
| device_id | string | Any valid OpenVINO device ID | string | Selects a particular hardware device for inference. The list of valid OpenVINO device ID's available on a platform can be obtained either by Python API (`onnxruntime.capi._pybind_state.get_available_openvino_device_ids()`) or by [OpenVINO C/C++ API](https://docs.openvino.ai/latest/classInferenceEngine_1_1Core.html). If this option is not explicitly set, an arbitrary free device will be automatically selected by OpenVINO runtime.|
| enable_vpu_fast_compile | string | True/False | boolean | This option is only available for MYRIAD_FP16 VPU devices. During initialization of the VPU device with compiled model, Fast-compile may be optionally enabled to speeds up the model's compilation to VPU device specific format. This in-turn speeds up model initialization time. However, enabling this option may slowdown inference due to some of the optimizations not being fully applied, so caution is to be exercised while enabling this option. |
| num_of_threads | string | Any unsigned positive number other than 0 | size_t | Overrides the accelerator default value of number of threads with this value at runtime. If this option is not explicitly set, default value of 8 is used during build time. |
@ -293,7 +254,7 @@ The following table lists all the available configuration options and the Key-Va
Valid Hetero or Multi or Auto Device combinations:
HETERO:<DEVICE_TYPE_1>,<DEVICE_TYPE_2>,<DEVICE_TYPE_3>...
The <DEVICE_TYPE> can be any of these devices from this list ['CPU','GPU','MYRIAD','FPGA','HDDL']
The <DEVICE_TYPE> can be any of these devices from this list ['CPU','GPU','MYRIAD']
A minimum of two DEVICE_TYPE'S should be specified for a valid HETERO or Multi-Device Build.
@ -346,141 +307,144 @@ Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_OpenVINO(sf, settings
**ONNX Layers supported using OpenVINO**
The table below shows the ONNX layers supported and validated using OpenVINO™ Execution Provider.The below table also lists the Intel hardware support for each of the layers. CPU refers to Intel<sup>®</sup>
Atom, Core, and Xeon processors. GPU refers to the Intel Integrated Graphics. VPU refers to USB based Intel<sup>®</sup> Movidius<sup>TM</sup>
VPUs as well as Intel<sup>®</sup> Vision accelerator Design with Intel Movidius <sup>TM</sup> MyriadX VPU.
Atom, Core, and Xeon processors. GPU refers to the Intel Integrated Graphics. Intel Discrete Graphics
| **ONNX Layers** | **CPU** | **GPU** | **VPU** |
| --- | --- | --- | --- |
| Abs | Yes | Yes | No |
| Acos | Yes | Yes | No |
| Acosh | Yes | Yes | No |
| Add | Yes | Yes | Yes |
| And | Yes | Yes | Yes |
| ArgMax | Yes | Yes | Yes |
| ArgMin | Yes | Yes | Yes |
| Asin | Yes | Yes | No |
| Asinh | Yes | Yes | No |
| Atan | Yes | Yes | No |
| Atanh | Yes | Yes | No |
| AveragePool | Yes | Yes | Yes |
| BatchNormalization | Yes | Yes | Yes |
| BitShift | Yes | No | No |
| Ceil | Yes | Yes | Yes |
| Celu | Yes | Yes | Yes |
| Cast | Yes | Yes | Yes |
| Clip | Yes | Yes | Yes |
| Concat | Yes | Yes | Yes |
| Constant | Yes | Yes | Yes |
| ConstantOfShape | Yes | Yes | Yes |
| Conv | Yes | Yes | Yes |
| ConvInteger | Yes | Yes | Yes |
| ConvTranspose | Yes | Yes | Yes |
| Cos | Yes | Yes | No |
| Cosh | Yes | Yes | No |
| CumSum | Yes | Yes | No |
| DepthToSpace | Yes | Yes | Yes |
| DequantizeLinear | Yes | Yes | No |
| Div | Yes | Yes | Yes |
| Dropout | Yes | Yes | Yes |
| Elu | Yes | Yes | Yes |
| Equal | Yes | Yes | Yes |
| Erf | Yes | Yes | Yes |
| Exp | Yes | Yes | Yes |
| Expand | Yes | Yes | Yes |
| EyeLike | Yes | No | No |
| Flatten | Yes | Yes | Yes |
| Floor | Yes | Yes | Yes |
| Gather | Yes | Yes | Yes |
| GatherElements | No | No | Yes |
| GatherND | Yes | Yes | Yes |
| Gemm | Yes | Yes | Yes |
| GlobalAveragePool | Yes | Yes | Yes |
| GlobalLpPool | Yes | Yes | No |
| GlobalMaxPool | Yes | Yes | No |
| **ONNX Layers** | **CPU** | **GPU** |
| --- | --- | --- |
| Abs | Yes | Yes |
| Acos | Yes | Yes |
| Acosh | Yes | Yes |
| Add | Yes | Yes |
| And | Yes | Yes |
| ArgMax | Yes | Yes |
| ArgMin | Yes | Yes |
| Asin | Yes | Yes |
| Asinh | Yes | Yes |
| Atan | Yes | Yes |
| Atanh | Yes | Yes |
| AveragePool | Yes | Yes |
| BatchNormalization | Yes | Yes |
| BitShift | Yes | No |
| Ceil | Yes | Yes |
| Celu | Yes | Yes |
| Cast | Yes | Yes |
| Clip | Yes | Yes |
| Concat | Yes | Yes |
| Constant | Yes | Yes |
| ConstantOfShape | Yes | Yes |
| Conv | Yes | Yes |
| ConvInteger | Yes | Yes |
| ConvTranspose | Yes | Yes |
| Cos | Yes | Yes |
| Cosh | Yes | Yes |
| CumSum | Yes | Yes |
| DepthToSpace | Yes | Yes |
| DequantizeLinear | Yes | Yes |
| Div | Yes | Yes |
| Dropout | Yes | Yes |
| Elu | Yes | Yes |
| Equal | Yes | Yes |
| Erf | Yes | Yes |
| Exp | Yes | Yes |
| Expand | Yes | Yes |
| EyeLike | Yes | No |
| Flatten | Yes | Yes |
| Floor | Yes | Yes |
| Gather | Yes | Yes |
| GatherElements | No | No |
| GatherND | Yes | Yes |
| Gemm | Yes | Yes |
| GlobalAveragePool | Yes | Yes |
| GlobalLpPool | Yes | Yes |
| GlobalMaxPool | Yes | Yes |
| Greater | Yes | Yes | Yes |
| GreaterOrEqual | Yes | Yes | Yes |
| HardMax | Yes | Yes | No |
| HardSigmoid | Yes | Yes | No |
| Identity | Yes | Yes | Yes |
| ImageScaler | Yes | Yes | Yes |
| InstanceNormalization | Yes | Yes | Yes |
| LeakyRelu | Yes | Yes | Yes |
| Less | Yes | Yes | Yes |
| LessOrEqual | Yes | Yes | Yes |
| Log | Yes | Yes | Yes |
| LogSoftMax | Yes | Yes | Yes |
| Loop | Yes | Yes | Yes |
| LRN | Yes | Yes | Yes |
| LSTM | Yes | Yes | Yes |
| MatMul | Yes | Yes | Yes |
| MatMulInteger | Yes | No | No |
| Max | Yes | Yes | Yes |
| MaxPool | Yes | Yes | Yes |
| Mean | Yes | Yes | Yes |
| MeanVarianceNormalization | Yes | Yes | Yes |
| Min | Yes | Yes | Yes |
| Mod | Yes | Yes | No |
| Mul | Yes | Yes | Yes |
| Neg | Yes | Yes | Yes |
| NonMaxSuppression | Yes | Yes | Yes |
| NonZero | Yes | No | Yes |
| Not | Yes | Yes | Yes |
| OneHot | Yes | Yes | Yes |
| Or | Yes | Yes | No |
| Pad | Yes | Yes | Yes |
| Pow | Yes | Yes | Yes |
| PRelu | Yes | Yes | Yes |
| QuantizeLinear | Yes | Yes | No |
| Range | Yes | Yes | Yes |
| Reciprocal | Yes | Yes | Yes |
| ReduceL1 | Yes | Yes | No |
| ReduceL2 | Yes | Yes | No |
| ReduceLogSum | Yes | Yes | Yes |
| ReduceLogSumExp | Yes | Yes | Yes |
| ReduceMax | Yes | Yes | Yes |
| ReduceMean | Yes | Yes | Yes |
| ReduceMin | Yes | Yes | Yes |
| ReduceProd | Yes | Yes | No |
| ReduceSum | Yes | Yes | Yes |
| ReduceSumSquare | Yes | Yes | Yes |
| Relu | Yes | Yes | Yes |
| Reshape | Yes | Yes | Yes |
| Resize | Yes | Yes | Yes |
| ReverseSequence | Yes | Yes | No |
| RoiAlign | Yes | Yes | Yes |
| Round | Yes | Yes | Yes |
| Scatter | Yes | Yes | Yes |
| ScatterElements | Yes | Yes | Yes |
| ScatterND | Yes | Yes | No |
| Selu | Yes | Yes | No |
| Shape | Yes | Yes | Yes |
| Shrink | Yes | Yes | No |
| Sigmoid | Yes | Yes | Yes |
| Sign | Yes | Yes | No |
| Sin | Yes | Yes | No |
| Sinh | Yes | No | No |
| SinFloat | No | No | Yes |
| Size | Yes | Yes | No |
| Slice | Yes | Yes | Yes |
| Softmax | Yes | Yes | Yes |
| Softplus | Yes | Yes | Yes |
| Softsign | Yes | Yes | Yes |
| SpaceToDepth | Yes | Yes | Yes |
| Split | Yes | Yes | Yes |
| Sqrt | Yes | Yes | Yes |
| Squeeze | Yes | Yes | Yes |
| Sub | Yes | Yes | Yes |
| Sum | Yes | Yes | Yes |
| Tan | Yes | Yes | No |
| Tanh | Yes | Yes | Yes |
| ThresholdedRelu | Yes | Yes | Yes |
| Tile | Yes | Yes | Yes |
| TopK | Yes | Yes | Yes |
| Transpose | Yes | Yes | Yes |
| Unsqueeze | Yes | Yes | Yes |
| Upsample | Yes | Yes | Yes |
| Where | Yes | Yes | Yes |
| Xor | Yes | Yes | No |
| GreaterOrEqual | Yes | Yes |
| GridSample | Yes | No |
| HardMax | Yes | Yes |
| HardSigmoid | Yes | Yes |
| Identity | Yes | Yes |
| If | Yes | Yes |
| ImageScaler | Yes | Yes |
| InstanceNormalization | Yes | Yes |
| LeakyRelu | Yes | Yes |
| Less | Yes | Yes |
| LessOrEqual | Yes | Yes |
| Log | Yes | Yes |
| LogSoftMax | Yes | Yes |
| Loop | Yes | Yes |
| LRN | Yes | Yes |
| LSTM | Yes | Yes |
| MatMul | Yes | Yes |
| MatMulInteger | Yes | No |
| Max | Yes | Yes |
| MaxPool | Yes | Yes |
| Mean | Yes | Yes |
| MeanVarianceNormalization | Yes | Yes |
| Min | Yes | Yes |
| Mod | Yes | Yes |
| Mul | Yes | Yes |
| Neg | Yes | Yes |
| NonMaxSuppression | Yes | Yes |
| NonZero | Yes | No |
| Not | Yes | Yes |
| OneHot | Yes | Yes |
| Or | Yes | Yes |
| Pad | Yes | Yes |
| Pow | Yes | Yes |
| PRelu | Yes | Yes |
| QuantizeLinear | Yes | Yes |
| QLinearMatMul | Yes | No |
| Range | Yes | Yes |
| Reciprocal | Yes | Yes |
| ReduceL1 | Yes | Yes |
| ReduceL2 | Yes | Yes |
| ReduceLogSum | Yes | Yes |
| ReduceLogSumExp | Yes | Yes |
| ReduceMax | Yes | Yes |
| ReduceMean | Yes | Yes |
| ReduceMin | Yes | Yes |
| ReduceProd | Yes | Yes |
| ReduceSum | Yes | Yes |
| ReduceSumSquare | Yes | Yes |
| Relu | Yes | Yes |
| Reshape | Yes | Yes |
| Resize | Yes | Yes |
| ReverseSequence | Yes | Yes |
| RoiAlign | Yes | Yes |
| Round | Yes | Yes |
| Scatter | Yes | Yes |
| ScatterElements | Yes | Yes |
| ScatterND | Yes | Yes |
| Selu | Yes | Yes |
| Shape | Yes | Yes |
| Shrink | Yes | Yes |
| Sigmoid | Yes | Yes |
| Sign | Yes | Yes |
| Sin | Yes | Yes |
| Sinh | Yes | No |
| SinFloat | No | No |
| Size | Yes | Yes |
| Slice | Yes | Yes |
| Softmax | Yes | Yes |
| Softplus | Yes | Yes |
| Softsign | Yes | Yes |
| SpaceToDepth | Yes | Yes |
| Split | Yes | Yes |
| Sqrt | Yes | Yes |
| Squeeze | Yes | Yes |
| Sub | Yes | Yes |
| Sum | Yes | Yes |
| Softsign | Yes | No |
| Tan | Yes | Yes |
| Tanh | Yes | Yes |
| ThresholdedRelu | Yes | Yes |
| Tile | Yes | Yes |
| TopK | Yes | Yes |
| Transpose | Yes | Yes |
| Unsqueeze | Yes | Yes |
| Upsample | Yes | Yes |
| Where | Yes | Yes |
| Xor | Yes | Yes |
### Topology Support
@ -489,86 +453,86 @@ Below topologies from ONNX open model zoo are fully supported on OpenVINO™ Exe
### Image Classification Networks
| **MODEL NAME** | **CPU** | **GPU** | **VPU** |
| --- | --- | --- | --- |
| bvlc_alexnet | Yes | Yes | Yes |
| bvlc_googlenet | Yes | Yes | Yes |
| bvlc_reference_caffenet | Yes | Yes | Yes |
| bvlc_reference_rcnn_ilsvrc13 | Yes | Yes | Yes |
| emotion ferplus | Yes | Yes | Yes |
| densenet121 | Yes | Yes | Yes |
| inception_v1 | Yes | Yes | Yes |
| inception_v2 | Yes | Yes | Yes |
| mobilenetv2 | Yes | Yes | Yes |
| resnet18v1 | Yes | Yes | Yes |
| resnet34v1 | Yes | Yes | Yes |
| resnet101v1 | Yes | Yes | Yes |
| resnet152v1 | Yes | Yes | Yes |
| resnet18v2 | Yes | Yes | Yes |
| resnet34v2 | Yes | Yes | Yes |
| resnet101v2 | Yes | Yes | Yes |
| resnet152v2 | Yes | Yes | Yes |
| resnet50 | Yes | Yes | Yes |
| resnet50v2 | Yes | Yes | Yes |
| shufflenet | Yes | Yes | Yes |
| squeezenet1.1 | Yes | Yes | Yes |
| vgg19 | Yes | Yes | Yes |
| vgg16 | Yes | Yes | Yes |
| zfnet512 | Yes | Yes | Yes |
| mxnet_arcface | No | Yes | No |
| **MODEL NAME** | **CPU** | **GPU** |
| --- | --- | --- |
| bvlc_alexnet | Yes | Yes |
| bvlc_googlenet | Yes | Yes |
| bvlc_reference_caffenet | Yes | Yes |
| bvlc_reference_rcnn_ilsvrc13 | Yes | Yes |
| emotion ferplus | Yes | Yes |
| densenet121 | Yes | Yes |
| inception_v1 | Yes | Yes |
| inception_v2 | Yes | Yes |
| mobilenetv2 | Yes | Yes |
| resnet18v1 | Yes | Yes |
| resnet34v1 | Yes | Yes |
| resnet101v1 | Yes | Yes |
| resnet152v1 | Yes | Yes |
| resnet18v2 | Yes | Yes |
| resnet34v2 | Yes | Yes |
| resnet101v2 | Yes | Yes |
| resnet152v2 | Yes | Yes |
| resnet50 | Yes | Yes |
| resnet50v2 | Yes | Yes |
| shufflenet | Yes | Yes |
| squeezenet1.1 | Yes | Yes |
| vgg19 | Yes | Yes |
| vgg16 | Yes | Yes |
| zfnet512 | Yes | Yes |
| mxnet_arcface | No | Yes |
### Image Recognition Networks
| **MODEL NAME** | **CPU** | **GPU** | **VPU** |
| --- | --- | --- | --- |
| mnist | Yes | Yes | Yes |
| **MODEL NAME** | **CPU** | **GPU** |
| --- | --- | --- |
| mnist | Yes | Yes |
### Object Detection Networks
| **MODEL NAME** | **CPU** | **GPU** | **VPU** |
| --- | --- | --- | --- |
| tiny_yolov2 | Yes | Yes | Yes |
| yolov3 | Yes | Yes | No |
| tiny_yolov3 | Yes | Yes | No |
| mask_rcnn | Yes | Yes | No |
| faster_rcnn | Yes | Yes | No |
| yolov4 | Yes | Yes | Yes |
| yolov5 | Yes | Yes | Yes |
| yolov7 | Yes | Yes | No |
| tiny_yolov7 | Yes | Yes | No |
| **MODEL NAME** | **CPU** | **GPU** |
| --- | --- | --- |
| tiny_yolov2 | Yes | Yes |
| yolov3 | Yes | Yes |
| tiny_yolov3 | Yes | Yes |
| mask_rcnn | Yes | Yes |
| faster_rcnn | Yes | Yes |
| yolov4 | Yes | Yes |
| yolov5 | Yes | Yes |
| yolov7 | Yes | Yes |
| tiny_yolov7 | Yes | Yes |
### Image Manipulation Networks
| **MODEL NAME** | **CPU** | **GPU** | **VPU** |
| --- | --- | --- | --- |
| mosaic | Yes | Yes | Yes |
| candy | Yes | Yes | Yes |
| cgan | Yes | Yes | Yes |
| rain_princess | Yes | yes | Yes |
| pointilism | Yes | Yes | Yes |
| udnie | Yes | Yes | Yes |
| **MODEL NAME** | **CPU** | **GPU** |
| --- | --- | --- |
| mosaic | Yes | Yes |
| candy | Yes | Yes |
| cgan | Yes | Yes |
| rain_princess | Yes | Yes |
| pointilism | Yes | Yes |
| udnie | Yes | Yes |
### Natural Language Processing Networks
| **MODEL NAME** | **CPU** | **GPU** | **VPU** |
| --- | --- | --- | --- |
| bert-squad | Yes | Yes | Yes |
| bert-base-cased | Yes | Yes | Yes |
| bert-base-chinese | Yes | Yes | Yes |
| bert-base-japanese-char | Yes | Yes | Yes |
| bert-base-multilingual-cased | Yes | yes | Yes |
| bert-base-uncased | Yes | Yes | Yes |
| distilbert-base-cased | Yes | Yes | No |
| distilbert-base-multilingual-cased | Yes | Yes | No |
| distilbert-base-uncased | Yes | Yes | No |
| distilbert-base-uncased-finetuned-sst-2-english | Yes | Yes | No |
| gpt2 | Yes | Yes | Yes |
| roberta-base | Yes | Yes | Yes |
| roberta-base-squad2 | Yes | Yes | Yes |
| t5-base | Yes | Yes | Yes |
| twitter-roberta-base-sentiment | Yes | Yes | Yes |
| xlm-roberta-base | Yes | Yes | Yes |
| **MODEL NAME** | **CPU** | **GPU** |
| --- | --- | --- |
| bert-squad | Yes | Yes |
| bert-base-cased | Yes | Yes |
| bert-base-chinese | Yes | Yes |
| bert-base-japanese-char | Yes | Yes |
| bert-base-multilingual-cased | Yes | Yes |
| bert-base-uncased | Yes | Yes |
| distilbert-base-cased | Yes | Yes |
| distilbert-base-multilingual-cased | Yes | Yes |
| distilbert-base-uncased | Yes | Yes |
| distilbert-base-uncased-finetuned-sst-2-english | Yes | Yes |
| gpt2 | Yes | Yes |
| roberta-base | Yes | Yes |
| roberta-base-squad2 | Yes | Yes |
| t5-base | Yes | Yes |
| twitter-roberta-base-sentiment | Yes | Yes |
| xlm-roberta-base | Yes | Yes |
**Note:** We have added support for INT8 models, quantized with Neural Network Compression Framework (NNCF). To know more about NNCF refer [here](https://github.com/openvinotoolkit/nncf).