saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-30 20:18:08 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Revert "Merge branch 'gh-pages' of https://github.com/microsoft/onnxruntime into gh-pages"

Browse source 
This reverts commit 063f84744e, reversing
changes made to 6f2f1e8dac.
This commit is contained in:
Prasanth Pulavarthi 2023-03-10 16:58:21 -08:00
parent 063f84744e
commit 20c745c749
41 changed files with 554 additions and 1329 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/execution-providers/OpenVINO-ExecutionProvider.md
View file

@ -170,7 +170,7 @@ Use the ONNX API's.[documentation](https://github.com/onnx/onnx/blob/master/docs
Example:
```python
```bash
import onnx
onnx_model = onnx.load("model.onnx") # Your model in memory as ModelProto
onnx.save_model(onnx_model, 'saved_model.onnx', save_as_external_data=True, all_tensors_to_one_file=True, location='data/weights_data', size_threshold=1024, convert_attribute=False)

170
docs/execution-providers/TensorRT-ExecutionProvider.md
View file

@ -26,8 +26,7 @@ Pre-built packages and Docker images are available for Jetpack in the [Jetson Zo
|ONNX Runtime|TensorRT|CUDA|
|---|---|---|
|main|8.5|11.6|
|1.14|8.5|11.6|
|main|8.5|11.4|
|1.12-1.13|8.4|11.4|
|1.11|8.2|11.4|
|1.10|8.0|11.4|
@ -43,11 +42,11 @@ For more details on CUDA/cuDNN versions, please see [CUDA EP requirements](./CUD
See [Build instructions](../build/eps.md#tensorrt).
The TensorRT execution provider for ONNX Runtime is built and tested with TensorRT 8.5.
The TensorRT execution provider for ONNX Runtime is built and tested with TensorRT 8.4.
## Usage
### C/C++
```c++
```
Ort::Env env = Ort::Env{ORT_LOGGING_LEVEL_ERROR, "Default"};
Ort::SessionOptions sf;
int device_id = 0;
@ -58,14 +57,14 @@ Ort::Session session(env, model_path, sf);
The C API details are [here](../get-started/with-c.md).
### Shape Inference for TensorRT Subgraphs
If some operators in the model are not supported by TensorRT, ONNX Runtime will partition the graph and only send supported subgraphs to TensorRT execution provider. Because TensorRT requires that all inputs of the subgraphs have shape specified, ONNX Runtime will throw error if there is no input shape info. In this case please run shape inference for the entire model first by running script [here](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/symbolic_shape_infer.py) (Check below for sample).
#### Shape Inference for TensorRT Subgraphs
If some operators in the model are not supported by TensorRT, ONNX Runtime will partition the graph and only send supported subgraphs to TensorRT execution provider. Because TensorRT requires that all inputs of the subgraphs have shape specified, ONNX Runtime will throw error if there is no input shape info. In this case please run shape inference for the entire model first by running script [here](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/symbolic_shape_infer.py).
### Python
To use TensorRT execution provider, you must explicitly register TensorRT execution provider when instantiating the `InferenceSession`.
Note that it is recommended you also register `CUDAExecutionProvider` to allow Onnx Runtime to assign nodes to CUDA execution provider that TensorRT does not support.
```python
To use TensorRT execution provider, you must explicitly register TensorRT execution provider when instantiating the InferenceSession.
Note that it is recommended you also register CUDAExecutionProvider to allow Onnx Runtime to assign nodes to CUDA execution provider that TensorRT does not support.
```
import onnxruntime as ort
# set providers to ['TensorrtExecutionProvider', 'CUDAExecutionProvider'] with TensorrtExecutionProvider having the higher priority.
sess = ort.InferenceSession('model.onnx', providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider'])
@ -77,110 +76,105 @@ There are two ways to configure TensorRT settings, either by environment variabl
### Environment Variables
Following environment variables can be set for TensorRT execution provider.
* `ORT_TENSORRT_MAX_WORKSPACE_SIZE`: maximum workspace size for TensorRT engine. Default value: 1073741824 (1GB).
* ORT_TENSORRT_MAX_WORKSPACE_SIZE: maximum workspace size for TensorRT engine. Default value: 1073741824 (1GB).
* `ORT_TENSORRT_MAX_PARTITION_ITERATIONS`: maximum number of iterations allowed in model partitioning for TensorRT. If target model can't be successfully partitioned when the maximum number of iterations is reached, the whole model will fall back to other execution providers such as CUDA or CPU. Default value: 1000.
* ORT_TENSORRT_MAX_PARTITION_ITERATIONS: maximum number of iterations allowed in model partitioning for TensorRT. If target model can't be successfully partitioned when the maximum number of iterations is reached, the whole model will fall back to other execution providers such as CUDA or CPU. Default value: 1000.
* `ORT_TENSORRT_MIN_SUBGRAPH_SIZE`: minimum node size in a subgraph after partitioning. Subgraphs with smaller size will fall back to other execution providers. Default value: 1.
* ORT_TENSORRT_MIN_SUBGRAPH_SIZE: minimum node size in a subgraph after partitioning. Subgraphs with smaller size will fall back to other execution providers. Default value: 1.
* `ORT_TENSORRT_FP16_ENABLE`: Enable FP16 mode in TensorRT. 1: enabled, 0: disabled. Default value: 0. Note not all Nvidia GPUs support FP16 precision.
* ORT_TENSORRT_FP16_ENABLE: Enable FP16 mode in TensorRT. 1: enabled, 0: disabled. Default value: 0. Note not all Nvidia GPUs support FP16 precision.
* `ORT_TENSORRT_INT8_ENABLE`: Enable INT8 mode in TensorRT. 1: enabled, 0: disabled. Default value: 0. Note not all Nvidia GPUs support INT8 precision.
* ORT_TENSORRT_INT8_ENABLE: Enable INT8 mode in TensorRT. 1: enabled, 0: disabled. Default value: 0. Note not all Nvidia GPUs support INT8 precision.
* `ORT_TENSORRT_INT8_CALIBRATION_TABLE_NAME`: Specify INT8 calibration table file for non-QDQ models in INT8 mode. Note calibration table should not be provided for QDQ model because TensorRT doesn't allow calibration table to be loded if there is any Q/DQ node in the model. By default the name is empty.
* ORT_TENSORRT_INT8_CALIBRATION_TABLE_NAME: Specify INT8 calibration table file for non-QDQ models in INT8 mode. Note calibration table should not be provided for QDQ model because TensorRT doesn't allow calibration table to be loded if there is any Q/DQ node in the model. By default the name is empty.
* `ORT_TENSORRT_INT8_USE_NATIVE_CALIBRATION_TABLE`: Select what calibration table is used for non-QDQ models in INT8 mode. If 1, native TensorRT generated calibration table is used; if 0, ONNXRUNTIME tool generated calibration table is used. Default value: 0.
* **Note: Please copy up-to-date calibration table file to `ORT_TENSORRT_CACHE_PATH` before inference. Calibration table is specific to models and calibration data sets. Whenever new calibration table is generated, old file in the path should be cleaned up or be replaced.**
* ORT_TENSORRT_INT8_USE_NATIVE_CALIBRATION_TABLE: Select what calibration table is used for non-QDQ models in INT8 mode. If 1, native TensorRT generated calibration table is used; if 0, ONNXRUNTIME tool generated calibration table is used. Default value: 0.
**Note: Please copy up-to-date calibration table file to ORT_TENSORRT_CACHE_PATH before inference. Calibration table is specific to models and calibration data sets. Whenever new calibration table is generated, old file in the path should be cleaned up or be replaced.
* `ORT_TENSORRT_DLA_ENABLE`: Enable DLA (Deep Learning Accelerator). 1: enabled, 0: disabled. Default value: 0. Note not all Nvidia GPUs support DLA.
* ORT_TENSORRT_DLA_ENABLE: Enable DLA (Deep Learning Accelerator). 1: enabled, 0: disabled. Default value: 0. Note not all Nvidia GPUs support DLA.
* `ORT_TENSORRT_DLA_CORE`: Specify DLA core to execute on. Default value: 0.
* ORT_TENSORRT_DLA_CORE: Specify DLA core to execute on. Default value: 0.
* ORT_TENSORRT_ENGINE_CACHE_ENABLE: Enable TensorRT engine caching. The purpose of using engine caching is to save engine build time in the case that TensorRT may take long time to optimize and build engine. Engine will be cached when it's built for the first time so next time when new inference session is created the engine can be loaded directly from cache. In order to validate that the loaded engine is usable for current inference, engine profile is also cached and loaded along with engine. If current input shapes are in the range of the engine profile, the loaded engine can be safely used. Otherwise if input shapes are out of range, profile cache will be updated to cover the new shape and engine will be recreated based on the new profile (and also refreshed in the engine cache). Note each engine is created for specific settings such as model path/name, precision (FP32/FP16/INT8 etc), workspace, profiles etc, and specific GPUs and it's not portable, so it's essential to make sure those settings are not changing, otherwise the engine needs to be rebuilt and cached again. 1: enabled, 0: disabled. Default value: 0.
**Warning: Please clean up any old engine and profile cache files (.engine and .profile) if any of the following changes:**
- Model changes (if there are any changes to the model topology, opset version, operators etc.)
- ORT version changes (i.e. moving from ORT version 1.8 to 1.9)
- TensorRT version changes (i.e. moving from TensorRT 7.0 to 8.0)
- Hardware changes. (Engine and profile files are not portable and optimized for specific Nvidia hardware)
* `ORT_TENSORRT_ENGINE_CACHE_ENABLE`: Enable TensorRT engine caching. The purpose of using engine caching is to save engine build time in the case that TensorRT may take long time to optimize and build engine. Engine will be cached when it's built for the first time so next time when new inference session is created the engine can be loaded directly from cache. In order to validate that the loaded engine is usable for current inference, engine profile is also cached and loaded along with engine. If current input shapes are in the range of the engine profile, the loaded engine can be safely used. Otherwise if input shapes are out of range, profile cache will be updated to cover the new shape and engine will be recreated based on the new profile (and also refreshed in the engine cache). Note each engine is created for specific settings such as model path/name, precision (FP32/FP16/INT8 etc), workspace, profiles etc, and specific GPUs and it's not portable, so it's essential to make sure those settings are not changing, otherwise the engine needs to be rebuilt and cached again. 1: enabled, 0: disabled. Default value: 0.
* **Warning: Please clean up any old engine and profile cache files (.engine and .profile) if any of the following changes:**
* Model changes (if there are any changes to the model topology, opset version, operators etc.)
* ORT version changes (i.e. moving from ORT version 1.8 to 1.9)
* TensorRT version changes (i.e. moving from TensorRT 7.0 to 8.0)
* Hardware changes. (Engine and profile files are not portable and optimized for specific Nvidia hardware)
* ORT_TENSORRT_CACHE_PATH: Specify path for TensorRT engine and profile files if ORT_TENSORRT_ENGINE_CACHE_ENABLE is 1, or path for INT8 calibration table file if ORT_TENSORRT_INT8_ENABLE is 1.
* `ORT_TENSORRT_CACHE_PATH`: Specify path for TensorRT engine and profile files if `ORT_TENSORRT_ENGINE_CACHE_ENABLE` is 1, or path for INT8 calibration table file if ORT_TENSORRT_INT8_ENABLE is 1.
* ORT_TENSORRT_DUMP_SUBGRAPHS: Dumps the subgraphs that are transformed into TRT engines in onnx format to the filesystem. This can help debugging subgraphs, e.g. by using `trtexec --onnx my_model.onnx` and check the outputs of the parser. 1: enabled, 0: disabled. Default value: 0.
* `ORT_TENSORRT_DUMP_SUBGRAPHS`: Dumps the subgraphs that are transformed into TRT engines in onnx format to the filesystem. This can help debugging subgraphs, e.g. by using `trtexec --onnx my_model.onnx` and check the outputs of the parser. 1: enabled, 0: disabled. Default value: 0.
* ORT_TENSORRT_FORCE_SEQUENTIAL_ENGINE_BUILD: Sequentially build TensorRT engines across provider instances in multi-GPU environment. 1: enabled, 0: disabled. Default value: 0.
* `ORT_TENSORRT_FORCE_SEQUENTIAL_ENGINE_BUILD`: Sequentially build TensorRT engines across provider instances in multi-GPU environment. 1: enabled, 0: disabled. Default value: 0.
One can override default values by setting environment variables ORT_TENSORRT_MAX_WORKSPACE_SIZE, ORT_TENSORRT_MAX_PARTITION_ITERATIONS, ORT_TENSORRT_MIN_SUBGRAPH_SIZE, ORT_TENSORRT_FP16_ENABLE, ORT_TENSORRT_INT8_ENABLE, ORT_TENSORRT_INT8_CALIBRATION_TABLE_NAME, ORT_TENSORRT_INT8_USE_NATIVE_CALIBRATION_TABLE, ORT_TENSORRT_ENGINE_CACHE_ENABLE, ORT_TENSORRT_CACHE_PATH and ORT_TENSORRT_DUMP_SUBGRAPHS.
e.g. on Linux
* `ORT_TENSORRT_CONTEXT_MEMORY_SHARING_ENABLE`: Share execution context memory between TensorRT subgraphs. Default 0 = false, nonzero = true.
One can override default values by setting environment variables. e.g. on Linux:
```bash
# Override default max workspace size to 2GB
#### Override default max workspace size to 2GB
export ORT_TENSORRT_MAX_WORKSPACE_SIZE=2147483648
# Override default maximum number of iterations to 10
#### Override default maximum number of iterations to 10
export ORT_TENSORRT_MAX_PARTITION_ITERATIONS=10
# Override default minimum subgraph node size to 5
#### Override default minimum subgraph node size to 5
export ORT_TENSORRT_MIN_SUBGRAPH_SIZE=5
# Enable FP16 mode in TensorRT
#### Enable FP16 mode in TensorRT
export ORT_TENSORRT_FP16_ENABLE=1
# Enable INT8 mode in TensorRT
#### Enable INT8 mode in TensorRT
export ORT_TENSORRT_INT8_ENABLE=1
# Use native TensorRT calibration table
#### Use native TensorRT calibration table
export ORT_TENSORRT_INT8_USE_NATIVE_CALIBRATION_TABLE=1
# Enable TensorRT engine caching
#### Enable TensorRT engine caching
export ORT_TENSORRT_ENGINE_CACHE_ENABLE=1
# Please Note warning above. This feature is experimental.
# Engine cache files must be invalidated if there are any changes to the model, ORT version, TensorRT version or if the underlying hardware changes. Engine files are not portable across devices.
* Please Note warning above. This feature is experimental. Engine cache files must be invalidated if there are any changes to the model, ORT version, TensorRT version or if the
underlying hardware changes. Engine files are not portable across devices.
# Specify TensorRT cache path
#### Specify TensorRT cache path
export ORT_TENSORRT_CACHE_PATH="/path/to/cache"
# Dump out subgraphs to run on TensorRT
export ORT_TENSORRT_DUMP_SUBGRAPHS=1
# Enable context memory sharing between TensorRT subgraphs. Default 0 = false, nonzero = true
export ORT_TENSORRT_CONTEXT_MEMORY_SHARING_ENABLE=1
```
#### Dump out subgraphs to run on TensorRT
export ORT_TENSORRT_DUMP_SUBGRAPHS = 1
### Execution Provider Options
TensorRT configurations can also be set by execution provider option APIs. It's useful when each model and inference session have their own configurations. In this case, execution provider option settings will override any environment variable settings. All configurations should be set explicitly, otherwise default value will be taken. There are one-to-one mappings between environment variables and execution provider options shown as below,
TensorRT configurations can also be set by execution provider option APIs. It's useful when each model and inference session have their own configurations. In this case, execution provider option settings will override any environment variable settings. All configurations should be set explicitly, otherwise default value will be taken.
ORT_TENSORRT_MAX_WORKSPACE_SIZE <-> trt_max_workspace_size
There are one-to-one mappings between **environment variables** and **execution provider options APIs** shown as below:
ORT_TENSORRT_MAX_PARTITION_ITERATIONS <-> trt_max_partition_iterations
> Note: for bool type options, assign them with **True**/**False** in python, or **1**/**0** in C++.
ORT_TENSORRT_MIN_SUBGRAPH_SIZE <-> trt_min_subgraph_size
| environment variables | execution provider option APIs | type |
| ---------------------------------------------- | ------------------------------------- | ------ |
| ORT_TENSORRT_MAX_WORKSPACE_SIZE | trt_max_workspace_size | int |
| ORT_TENSORRT_MAX_PARTITION_ITERATIONS | trt_max_partition_iterations | int |
| ORT_TENSORRT_MIN_SUBGRAPH_SIZE | trt_min_subgraph_size | int |
| ORT_TENSORRT_FP16_ENABLE | trt_fp16_enable | bool |
| ORT_TENSORRT_INT8_ENABLE | trt_int8_enable | bool |
| ORT_TENSORRT_INT8_CALIBRATION_TABLE_NAME | trt_int8_calibration_table_name | string |
| ORT_TENSORRT_INT8_USE_NATIVE_CALIBRATION_TABLE | trt_int8_use_native_calibration_table | bool |
| ORT_TENSORRT_DLA_ENABLE | trt_dla_enable | bool |
| ORT_TENSORRT_DLA_CORE | trt_dla_core | int |
| ORT_TENSORRT_ENGINE_CACHE_ENABLE | trt_engine_cache_enable | bool |
| ORT_TENSORRT_CACHE_PATH | trt_engine_cache_path | string |
| ORT_TENSORRT_DUMP_SUBGRAPHS | trt_dump_subgraphs | bool |
| ORT_TENSORRT_FORCE_SEQUENTIAL_ENGINE_BUILD | trt_force_sequential_engine_build | bool |
| ORT_TENSORRT_CONTEXT_MEMORY_SHARING_ENABLE | trt_context_memory_sharing_enable | bool |
ORT_TENSORRT_FP16_ENABLE <-> trt_fp16_enable
Besides, `device_id` can also be set by execution provider option.
ORT_TENSORRT_INT8_ENABLE <-> trt_int8_enable
ORT_TENSORRT_INT8_CALIBRATION_TABLE_NAME <-> trt_int8_calibration_table_name
ORT_TENSORRT_INT8_USE_NATIVE_CALIBRATION_TABLE <-> trt_int8_use_native_calibration_table
ORT_TENSORRT_DLA_ENABLE <-> trt_dla_enable
ORT_TENSORRT_DLA_CORE <-> trt_dla_core
ORT_TENSORRT_ENGINE_CACHE_ENABLE <-> trt_engine_cache_enable
ORT_TENSORRT_CACHE_PATH <-> trt_engine_cache_path
ORT_TENSORRT_DUMP_SUBGRAPHS <-> trt_dump_subgraphs
ORT_TENSORRT_FORCE_SEQUENTIAL_ENGINE_BUILD <-> trt_force_sequential_engine_build
Besides, device_id can also be set by execution provider option.
#### C++ API example
```c++
```
Ort::SessionOptions session_options;
OrtTensorRTProviderOptions trt_options{};
// note: for bool type options in c++ API, set them as 0/1
trt_options.device_id = 1;
trt_options.trt_max_workspace_size = 2147483648;
trt_options.trt_max_partition_iterations = 10;
@ -195,20 +189,19 @@ session_options.AppendExecutionProvider_TensorRT(trt_options);
```
#### Python API example
```python
```
import onnxruntime as ort
model_path = '<path to model>'
# note: for bool type options in python API, set them as False/True
providers = [
('TensorrtExecutionProvider', {
'device_id': 0,
'device_id': 1,
'trt_max_workspace_size': 2147483648,
'trt_fp16_enable': True,
}),
('CUDAExecutionProvider', {
'device_id': 0,
'device_id': 1,
'arena_extend_strategy': 'kNextPowerOfTwo',
'gpu_mem_limit': 2 * 1024 * 1024 * 1024,
'cudnn_conv_algo_search': 'EXHAUSTIVE',
@ -223,8 +216,7 @@ sess = ort.InferenceSession(model_path, sess_options=sess_opt, providers=provide
## Performance Tuning
For performance tuning, please see guidance on this page: [ONNX Runtime Perf Tuning](../performance/tune-performance.md)
When/if using [onnxruntime_perf_test](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/perftest#onnxruntime-performance-test), use the flag `-e tensorrt`. Check below for sample.
When/if using [onnxruntime_perf_test](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/perftest#onnxruntime-performance-test), use the flag `-e tensorrt`
## Samples
@ -233,26 +225,14 @@ This example shows how to run the Faster R-CNN model on TensorRT execution provi
1. Download the Faster R-CNN onnx model from the ONNX model zoo [here](https://github.com/onnx/models/tree/master/vision/object_detection_segmentation/faster-rcnn).
2. Infer shapes in the model by running the [shape inference script](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/symbolic_shape_infer.py)
```bash
```
python symbolic_shape_infer.py --input /path/to/onnx/model/model.onnx --output /path/to/onnx/model/new_model.onnx --auto_merge
```
3. Replace the original model with the new model and run the
onnx_test_runner tool under ONNX Runtime build directory.
```bash
./onnx_test_runner -e tensorrt /path/to/onnx/model/
```
4. Run `onnxruntime_perf_test` on your shape-inferred Faster-RCNN model
> Download sample test data with model from [model zoo](https://github.com/onnx/models/tree/main/vision/object_detection_segmentation/faster-rcnn), and put test_data_set folder next to your inferred model
```bash
# e.g.
# -r: set up test repeat time
# -e: set up execution provider
# -i: set up params for execution provider options
./onnxruntime_perf_test -r 1 -e tensorrt -i "trt_fp16_enable|true" /path/to/onnx/your_inferred_model.onnx
./onnx_test_runner -e tensorrt /path/to/onnx/model/
```
Please see [this Notebook](https://github.com/microsoft/onnxruntime/blob/main/docs/python/inference/notebooks/onnx-inference-byoc-gpu-cpu-aks.ipynb) for an example of running a model on GPU using ONNX Runtime through Azure Machine Learning Services.

2
docs/get-started/community-projects.md
View file

@ -2,7 +2,7 @@
title: Julia and Ruby APIs
description: Get started with APIs for Julia and Ruby developed by our community
parent: Get Started
nav_order: 8
nav_order: 9
has_toc: false
---

2
docs/get-started/training-pytorch.md
View file

@ -1,7 +1,7 @@
---
title: ORT Training with PyTorch
parent: Get Started
nav_order: 12
nav_order: 10
---
# Get started with ORT for Training API (PyTorch)

4
docs/get-started/with-c.md
View file

@ -1,7 +1,7 @@
---
title: C
parent: Get Started
nav_order: 3
nav_order: 4
---
# Get started with ORT for C
@ -110,4 +110,4 @@ To turn on/off telemetry collection on official Windows builds, please use Enabl
## Samples
See [Candy Style Transfer](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_cxx/fns_candy_style_transfer)
See [Tutorials: API Basics - C](../tutorials/api-basics)

2
docs/get-started/with-csharp.md
View file

@ -2,7 +2,7 @@
title: C#
parent: Get Started
toc: true
nav_order: 4
nav_order: 3
---
# Get started with ORT for C#
{: .no_toc }

22
docs/get-started/with-mobile.md
View file

@ -1,22 +0,0 @@
---
title: Mobile
nav_order: 10
# manual TOC as we don't want some low level things to appear in it, but they need to be children of this page
# for the navigation to work as desired. e.g. doco on scripts mentioned in Model Export Helpers
parent: Get Started
has_toc: false
redirect_from: /docs/reference/mobile
---
# Get started with ONNX Runtime Mobile
ORT Mobile allows you to run model inferencing on mobile devices (iOS and Android).
## Reference
* [Install ONNX Runtime Mobile](./../install/index.md#install-on-web-and-mobile)
* [Tutorials: Deploy on mobile](./../tutorials/mobile/index.md)
* Build from source: [Android](./../build/android.md) / [iOS](./../build/ios.md)
* [ORT Mobile Operators](./../reference/operators/MobileOps.md)
* [Model Export Helpers](./../tutorials/mobile/helpers/index.md)
* [ORT Format Model Runtime Optimization](./../performance/ort-format-model-runtime-optimization.md)

20
docs/get-started/with-web.md
View file

@ -1,20 +0,0 @@
---
title: Web
description: ONNX Runtime for web-based deployments and considerations and options for building a web application with ONNX Runtime
has_toc: false
nav_order: 11
parent: Get Started
redirect_from: /docs/reference/build-web-app
---
# Get started with ONNX Runtime Web
ORT Web can be used in your web applications for model inferencing.
{: .no_toc}
## Reference
* [Install ONNX Runtime Web](./../install/index.md#install-on-web-and-mobile)
* [Build from source](./../build/web.md)
* [Tutorials: Deploy on web](./../tutorials/web/index.md)
* [Guide: Build a web application with ONNX Runtime](./../tutorials/web/build-web-app)

2
docs/get-started/with-windows.md
View file

@ -3,7 +3,7 @@ title: Windows
parent: Get Started
toc: true
description: Get Started with Onnx Runtime with Windows. Windows OS Integration and requirements to install and build ORT for Windows are given.
nav_order: 9
nav_order: 8
---

2
docs/index.md
View file

@ -11,7 +11,7 @@ redirect_from: /how-to
ONNX Runtime is a cross-platform machine-learning model accelerator, with a flexible interface to integrate hardware-specific libraries. ONNX Runtime can be used with models from PyTorch, Tensorflow/Keras, TFLite, scikit-learn, and other frameworks.
<iframe height="315" class="table-wrapper py px" src="https://www.youtube.com/embed/waIeC3OIn70" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
<iframe height="315" class="table-wrapper py px" src="https://www.youtube.com/embed/vo9vlR-TRK4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
## How to use ONNX Runtime

4
docs/performance/graph-optimizations.md
View file

@ -135,11 +135,11 @@ session = rt.InferenceSession("<model_path>", sess_options)
g_ort->SetSessionGraphOptimizationLevel(session_options, ORT_ENABLE_EXTENDED);
// To enable model serialization after graph optimization set this
const ORTCHAR_T* optimized_model_path = ORT_TSTR("optimized_model_path");
const wchar_t* optimized_model_path = L"optimized_model_path";
g_ort->SetOptimizedModelFilePath(session_options, optimized_model_path);
OrtSession* session;
const ORTCHAR_T* model_path = ORT_TSTR("model_path");
const wchar_t* model_path = L"model_path";
g_ort->CreateSession(env, model_path, session_option, &session);
```

266
docs/performance/transformers-optimization.md
View file

@ -1,266 +0,0 @@
---
title: Transformers Optimizer
description: Transformer model optimization tool to use with ONNX Runtime
parent: Performance
nav_order: 4
---
# Transformers Optimizer
{: .no_toc }
## Contents
{: .no_toc }
* TOC placeholder
{:toc}
## Transformer Model Optimization Tool Overview
While ONNX Runtime automatically applies most optimizations while loading transformer models, some of the latest optimizations that have not yet been integrated into ONNX Runtime. These additional optimizations can be applied using a [separate tool](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/python/tools/transformers) that tunes models for the best performance. This optimization tool provides an offline capability to optimize transformer models in scenarios where ONNX Runtime does not apply the optimization at load time.
This tool can be helpful when:
* ONNX Runtime does not yet have transformer-specific graph optimization enabled
* The model can be converted to use float16 to boost performance using mixed precision on GPUs with Tensor Cores (like V100 or T4)
* The model has inputs with dynamic axis, which blocks some optimizations from being applied by ONNX Runtime due to shape inference.
* Experimenting with disabling or enabling some fusions to evaluate impact on performance or accuracy.
### Usage workflow
1. [Install ONNX Runtime](#1-install-onnx-runtime)
2. [Convert the transformer model to ONNX](#2-convert-a-transformer-model-to-onnx)
3. [Run the model optimizer tool](#3-run-the-model-optimizer-tool)
4. [Benchmark and profile the model](#4-benchmark-and-profile-the-model)
### Supported models
Below is the list of models that have been explicitly tested with the optimizer. Models not in the list may only be partially optimized or not optimized at all.
PyTorch (from [Huggingface Transformers](https://github.com/huggingface/transformers/)):
- BERT
- DistilBERT
- DistilGPT2
- RoBERTa
- ALBERT
- GPT-2 (**GPT2Model**, **GPT2LMHeadModel**)
TensorFlow
- The only TensorFlow model tested so far is BERT.
Most optimizations require exact match of a subgraph. Any layout change in the subgraph might cause some optimization to not work. Note that different versions of training or export tool might lead to different graph layouts. It is recommended to use the latest released version of PyTorch and Transformers.
#### Limitations to note
* Due to the CUDA implementation of the Attention kernel in ONNX Runtime, the maximum number of attention heads is 1024.
* Normally, due to GPU memory constraints, the maximum supported sequence length is 4096 for Longformer and 1024 for other types of models.
---
## 1. Install ONNX Runtime
First you need install onnxruntime or onnxruntime-gpu package for CPU or GPU inference. To use onnxruntime-gpu, it is required to install CUDA and cuDNN and add their bin directories to PATH environment variable. See [Python installation instructions](./../install/index.md#python-installs).
## 2. Convert a transformer model to ONNX
To convert the transformer model to ONNX, use [torch.onnx](https://pytorch.org/docs/stable/onnx.html) or [tensorflow-onnx](https://github.com/onnx/tensorflow-onnx).
- Huggingface transformers has a [notebook](https://github.com/huggingface/notebooks/blob/master/examples/onnx-export.ipynb) shows an example of exporting a pretrained model to ONNX.
- For tf2onnx, please refer to this [BERT tutorial](https://github.com/onnx/tensorflow-onnx/blob/master/tutorials/BertTutorial.ipynb).
### GPT-2 Model conversion
Converting the GPT-2 model from PyTorch to ONNX is not straightforward when past state is used. The tool [convert_to_onnx](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/models/gpt2/convert_to_onnx.py) can help.
You can use commands like the following to convert a pre-trained PyTorch GPT-2 model to ONNX for given precision (float32, float16):
```
python -m onnxruntime.transformers.models.gpt2.convert_to_onnx -m gpt2 --model_class GPT2LMHeadModel --output gpt2.onnx -p fp32
python -m onnxruntime.transformers.models.gpt2.convert_to_onnx -m distilgpt2 --model_class GPT2LMHeadModel --output distilgpt2.onnx -p fp16 --use_gpu --optimize_onnx --auto_mixed_precision
```
The tool will also verify whether the ONNX model and corresponding PyTorch model generate the same outputs given the same random inputs.
### Longformer Model conversion
Requirement: Linux OS (e.g. Ubuntu 18.04 or 20.04) and a Python environment with PyTorch 1.9.* like the following:
```
conda create -n longformer python=3.8
conda activate longformer
pip install torch==1.9.1+cpu torchvision==0.10.1+cpu torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
pip install onnx transformers==4.18.0 onnxruntime numpy
```
Next, build the source of [torch extensions for Longformer ONNX exporting](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/torch_extensions) like the following:
```
cd onnxruntime/python/tools/transformers/models/longformer/torch_extensions
python setup.py install
```
It will generate a PyTorch extension file like "build/lib.linux-x86_64-3.8/longformer_attention.cpython-38-x86_64-linux-gnu.so" under the directory.
Finally, convert longformer model to ONNX model like the following:
```
cd ..
python convert_to_onnx.py -m longformer-base-4096
```
The exported ONNX model can only run on GPU right now.
## 3. Run the model optimizer tool
In your Python code, you can use the optimizer like the following:
```python
from onnxruntime.transformers import optimizer
optimized_model = optimizer.optimize_model("bert.onnx", model_type='bert', num_heads=12, hidden_size=768)
optimized_model.convert_float_to_float16()
optimized_model.save_model_to_file("bert_fp16.onnx")
```
You can also use command line. Example of optimizing a BERT-large model to use mixed precision (float16):
```console
python -m onnxruntime.transformers.optimizer --input bert_large.onnx --output bert_large_fp16.onnx --num_heads 16 --hidden_size 1024 --float16
```
You can also download the latest script files from [here](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/). Then run it like the following:
```console
python optimizer.py --input bert.onnx --output bert_opt.onnx --model_type bert
```
### Optimizer Options
- **input**: input model path
- **output**: output model path
- **model_type**: (*defaul: bert*)
There are 4 model types: *bert* (exported by PyTorch), *gpt2* (exported by PyTorch), and *bert_tf* (BERT exported by tf2onnx), *bert_keras* (BERT exported by keras2onnx) respectively.
- **num_heads**: (*default: 12*)
Number of attention heads. BERT-base and BERT-large has 12 and 16 respectively.
- **hidden_size**: (*default: 768*)
BERT-base and BERT-large has 768 and 1024 hidden nodes respectively.
- **input_int32**: (*optional*)
Exported model ususally uses int64 tensor as input. If this flag is specified, int32 tensors will be used as input, and it could avoid un-necessary Cast nodes and get better performance.
- **float16**: (*optional*)
By default, model uses float32 in computation. If this flag is specified, half-precision float will be used. This option is recommended for NVidia GPU with Tensor Core like V100 and T4. For older GPUs, float32 is likely faster.
- **use_gpu**: (*optional*)
When opt_level > 1, please set this flag for GPU inference.
- **opt_level**: (*optional*)
Set a proper graph optimization level of OnnxRuntime: 0 - disable all (default), 1 - basic, 2 - extended, 99 - all. If the value is positive, OnnxRuntime will be used to optimize graph first.
- **verbose**: (*optional*)
Print verbose information when this flag is specified.
### BERT Model Verification
If your BERT model has three inputs (like input_ids, token_type_ids and attention_mask), a script compare_bert_results.py can be used to do a quick verification. The tool will generate some fake input data, and compare results from both the original and optimized models. If outputs are all close, it is safe to use the optimized model.
Example of verifying models optimized for CPU:
```console
python -m onnxruntime.transformers.compare_bert_results --baseline_model original_model.onnx --optimized_model optimized_model_cpu.onnx --batch_size 1 --sequence_length 128 --samples 100
```
For GPU, please append --use_gpu to the command.
## 4. Benchmark and profile the model
### Benchmark
There is a bash script [run_benchmark.sh](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/run_benchmark.sh) for running benchmark. You can modify the bash script to choose your options (like models to test, batch sizes, sequence lengths, target device etc) before running.
The bash script will call benchmark.py script to measure inference performance of OnnxRuntime, PyTorch or PyTorch+TorchScript on pretrained models of Huggingface Transformers.
### Performance Test
bert_perf_test.py can be used to check the BERT model inference performance. Below are examples:
```console
python -m onnxruntime.transformers.bert_perf_test --model optimized_model_cpu.onnx --batch_size 1 --sequence_length 128
```
For GPU, please append --use_gpu to the command.
After test is finished, a file like perf_results_CPU_B1_S128_<date_time>.txt or perf_results_GPU_B1_S128_<date_time>.txt will be output to the model directory.
### Profiling
profiler.py can be used to run profiling on a transformer model. It can help figure out the bottleneck of a model, and CPU time spent on a node or subgraph.
Examples commands:
```console
python -m onnxruntime.transformers.profiler --model bert.onnx --batch_size 8 --sequence_length 128 --samples 1000 --dummy_inputs bert --thread_num 8 --kernel_time_only
python -m onnxruntime.transformers.profiler --model gpt2.onnx --batch_size 1 --sequence_length 1 --past_sequence_length 128 --samples 1000 --dummy_inputs gpt2 --use_gpu
python -m onnxruntime.transformers.profiler --model longformer.onnx --batch_size 1 --sequence_length 4096 --global_length 8 --samples 1000 --dummy_inputs longformer --use_gpu
```
Result file like onnxruntime_profile__<date_time>.json will be output to current directory. Summary of nodes, top expensive nodes and results grouped by operator type will be printed to console.
For benchmark results, see [Appendix](#appendix)
## Appendix
### Benchmark Results on V100
In the following benchmark results, ONNX Runtime uses optimizer for model optimization, and IO binding is enabled.
We tested on Tesla V100-PCIE-16GB GPU (CPU is Intel Xeon(R) E5-2690 v4) for different batch size (**b**) and sequence length (**s**). Below result is average latency of per inference in miliseconds.
#### bert-base-uncased (BertModel)
The model has 12 layers and 768 hidden, with input_ids as input.
| engine | version | precision | b | s=8 | s=16 | s=32 | s=64 | s=128 | s=256 | s=512 |
|-------------|---------|-----------|---|------|------|------|------|-------|-------|-------|
| torchscript | 1.5.1 | fp32 | 1 | 7.92 | 8.78 | 8.91 | 9.18 | 9.56 | 9.39 | 12.83 |
| onnxruntime | 1.4.0 | fp32 | 1 | 1.38 | 1.42 | 1.67 | 2.15 | 3.11 | 5.37 | 10.74 |
| onnxruntime | 1.4.0 | fp16 | 1 | 1.30 | 1.29 | 1.31 | 1.33 | 1.45 | 1.95 | 3.36 |
| onnxruntime | 1.4.0 | fp32 | 4 | 1.51 | 1.93 | 2.98 | 5.01 | 9.13 | 17.95 | 38.15 |
| onnxruntime | 1.4.0 | fp16 | 4 | 1.27 | 1.35 | 1.43 | 1.83 | 2.66 | 4.40 | 9.76 |
[run_benchmark.sh](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/run_benchmark.sh) is used to get the results.
#### gpt2 (GPT2LMHeadModel)
The model has 12 layers and 768 hidden, with input_ids, position_ids, attention_mask and past state as inputs.
| engine | version | precision | b | s=4 | s=8 | s=32 | s=128 |
|-------------|---------|-----------|---|------|------|------|------|
| torchscript | 1.5.1 | fp32 | 1 | 5.80 | 5.77 | 5.82 | 5.78 |
| onnxruntime | 1.4.0 | fp32 | 1 | 1.42 | 1.42 | 1.43 | 1.47 |
| onnxruntime | 1.4.0 | fp16 | 1 | 1.54 | 1.54 | 1.58 | 1.64 |
| onnxruntime | 1.4.0 | fp32 | 8 | 1.83 | 1.84 | 1.90 | 2.13 |
| onnxruntime | 1.4.0 | fp16 | 8 | 1.74 | 1.75 | 1.81 | 2.09 |
| onnxruntime | 1.4.0 | fp32 | 32 | 2.19 | 2.21 | 2.45 | 3.34 |
| onnxruntime | 1.4.0 | fp16 | 32 | 1.66 | 1.71 | 1.85 | 2.73 |
| onnxruntime | 1.4.0 | fp32 | 128 | 4.15 | 4.37 | 5.15 | 8.61 |
| onnxruntime | 1.4.0 | fp16 | 128 | 2.47 | 2.58 | 3.26 | 6.16 |
Since past state is used, sequence length in input_ids is 1. For example, s=4 means the past sequence length is 4 and the total sequence length is 5.
[benchmark_gpt2.py](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/models/gpt2/benchmark_gpt2.py) is used to get the results like the following commands:
```console
python -m onnxruntime.transformers.models.gpt2.benchmark_gpt2 --use_gpu -m gpt2 -o -v -b 1 8 32 128 -s 4 8 32 128 -p fp32
python -m onnxruntime.transformers.models.gpt2.benchmark_gpt2 --use_gpu -m gpt2 -o -v -b 1 8 32 128 -s 4 8 32 128 -p fp16
```
### Benchmark.py
If you use run_benchmark.sh, you need not use benchmark.py directly. You can skip this section if you do not want to know the details.
Below is example to running benchmark.py on pretrained model bert-base-cased on GPU.
```console
python -m onnxruntime.transformers.benchmark -g -m bert-base-cased -o -v -b 0
python -m onnxruntime.transformers.benchmark -g -m bert-base-cased -o
python -m onnxruntime.transformers.benchmark -g -m bert-base-cased -e torch
python -m onnxruntime.transformers.benchmark -g -m bert-base-cased -e torchscript
```
The first command will generate ONNX models (both before and after optimizations), but not run performance tests since batch size is 0. The other three commands will run performance test on each of three engines: OnnxRuntime, PyTorch and PyTorch+TorchScript.
If you remove -o parameter, optimizer script is not used in benchmark.
If your GPU (like V100 or T4) has TensorCore, you can append `-p fp16` to the above commands to enable mixed precision.
If you want to benchmark on CPU, you can remove -g option in the commands.
Note that our current benchmark on GPT2 and DistilGPT2 models has disabled past state from inputs and outputs.
By default, ONNX model has only one input (input_ids). You can use -i parameter to test models with multiple inputs. For example, we can add "-i 3" to command line to test a bert model with 3 inputs (input_ids, token_type_ids and attention_mask). This option only supports OnnxRuntime right now.

2
docs/performance/tune-performance.md
View file

@ -531,7 +531,7 @@ api.SessionOptionsAppendExecutionProvider_CUDA_V2(static_cast<OrtSessionOptions*
// Create IO bound inputs and outputs.
Ort::Session session(*ort_env, ORT_TSTR("matmul_2.onnx"), session_options);
Ort::Session session(*ort_env, L"matmul_2.onnx", session_options);
Ort::MemoryInfo info_cuda("Cuda", OrtAllocatorType::OrtArenaAllocator, 0, OrtMemTypeDefault);
Ort::Allocator cuda_allocator(session, info_cuda);

21
docs/tutorials/web/build-web-app.md → docs/reference/build-web-app.md
View file

@ -1,15 +1,13 @@
---
title: Build a web app with ONNX Runtime
description: Considerations and options for building a web application with ONNX Runtime
parent: Deploy on web
grand_parent: Tutorials
nav_order: 3
redirect_from: /reference/build-web-app
parent: Reference
has_children: false
nav_order: 6
---
# Build a web application with ONNX Runtime
{: .no_toc}
This document explains the options and considerations for building a web application with ONNX Runtime.
## Contents
{: .no_toc}
@ -17,6 +15,7 @@ This document explains the options and considerations for building a web applica
* TOC
{:toc}
This document explains the options and considerations for building a web application with ONNX Runtime.
## Options for deployment target
@ -48,9 +47,9 @@ You need to understand your web app's scenario and get an ONNX model that is app
ONNX models can be obtained from the [ONNX model zoo](https://github.com/onnx/models), converted from PyTorch or TensorFlow, and many other places.
You can [convert the ONNX format model to ORT format model](../../reference/ort-format-models.md), for optimized binary size, faster initialization and peak memory usage.
You can [convert the ONNX format model to ORT format model](./ort-format-models.md), for optimized binary size, faster initialization and peak memory usage.
You can [perform a model-specific custom build](../../build/custom.md) to further optimize binary size.
You can [perform a model-specific custom build](../build/custom.md) to further optimize binary size.
## Bootstrap your application
@ -79,7 +78,7 @@ Add "@dev" to the package name to use the nightly build (eg. npm install onnxrun
## Consume onnxruntime-web in your code
1. Import onnxruntime-web
See [import onnxruntime-web](../../get-started/with-javascript.md#import-1)
See [import onnxruntime-web](../get-started/with-javascript.md#import-1)
2. Initialize the inference session
See [InferenceSession.create](https://github.com/microsoft/onnxruntime-inference-examples/blob/main/js/quick-start_onnxruntime-web-bundler/main.js#L14)
@ -91,7 +90,7 @@ Add "@dev" to the package name to use the nightly build (eg. npm install onnxrun
Session run happens each time their is new user input.
Refer to [ONNX Runtime Web API docs](../../api/js) for more detail.
Refer to [ONNX Runtime Web API docs](../api/js) for more detail.
## Pre and post processing
@ -105,7 +104,7 @@ Raw input is usually a string (for NLP model) or an image (for image model). The
### Image input
1. Use a JS/wasm library to pre-process the data, and create tensor as input to fulfill the requirement of the model. See the [image classification using ONNX Runtime Web](./classify-images-nextjs-github-template.md) tutorial.
1. Use a JS/wasm library to pre-process the data, and create tensor as input to fulfill the requirement of the model. See the [image classification using ONNX Runtime Web](../tutorials/web/classify-images-nextjs-github-template.md) tutorial.
2. Modify the model to include the pre-processing inside the model as operators. The model will expect a certain web image format (eg. A bitmap or texture from canvas).

114
docs/reference/mobile/helpers.md Normal file
View file

@ -0,0 +1,114 @@
---
title: Model Export Helpers
descriptions: Helpers to assist with export and usage of models with ORT Mobile
parent: Mobile
grand_parent: Reference
nav_order: 1
---
# ORT Mobile Model Export Helpers
{: .no_toc }
There are a range of tools available to aid with exporting and analyzing a model for usage with ORT Mobile.
## Contents
{: .no_toc }
* TOC placeholder
{:toc}
## ORT Mobile model usability checker
The model usability checker provides information on how well a model is likely to run with ORT mobile, including the suitability for using NNAPI on Android and CoreML on iOS. It can also recommend running specific tools to update the model so that it works better with ORT Mobile.
See [here](./model-usability-checker.md) for more details.
## ONNX model opset updater
The ORT Mobile pre-built package only supports the most recent ONNX opsets in order to minimize binary size. Most ONNX models can be updated to a newer ONNX opset using this tool. It is recommended to use the latest opset the pre-built package supports, which is currently opset 15.
The ONNX opsets supported by the pre-built package are documented [here](../operators/mobile_package_op_type_support_1.10.md).
Usage:
```
python -m onnxruntime.tools.update_onnx_opset --help
usage: update_onnx_opset.py:update_onnx_opset_helper [-h] [--opset OPSET] input_model output_model
Update the ONNX opset of the model. New opset must be later than the existing one. If not specified will update to opset 15.
positional arguments:
input_model Provide path to ONNX model to update.
output_model Provide path to write updated ONNX model to.
optional arguments:
-h, --help show this help message and exit
--opset OPSET ONNX opset to update to.
```
Example usage:
```
python -m onnxruntime.tools.update_onnx_opset --opset 15 model.onnx model.opset15.onnx
```
## ONNX model dynamic shape fixer
If the model can potentially be used with NNAPI or CoreML it may require the input shapes to be made 'fixed' by setting any dynamic dimension sizes to specific values.
See [here](./make-dynamic-shape-fixed.md) for more details.
## QDQ format model helpers
Depending on the source of a QDQ format model, it may be necessary to optimize aspects of it to ensure optimal performance with ORT.
The onnxruntime.tools.qdq_helpers.optimize_qdq_model helper can be used to do this.
Usage:
```
python -m onnxruntime.tools.qdq_helpers.optimize_qdq_model --help
usage: optimize_qdq_model.py [-h] input_model output_model
Update a QDQ format ONNX model to ensure optimal performance when executed using ONNX Runtime.
positional arguments:
input_model Provide path to ONNX model to update.
output_model Provide path to write updated ONNX model to.
optional arguments:
-h, --help show this help message and exit
```
Note that if there are no optimizations the output_model will be the same as the input_model and can be discarded.
## PyTorch export helpers
When exporting a model from [PyTorch](https://pytorch.org/) using [torch.onnx.export](https://pytorch.org/docs/stable/onnx.html) the names of the model inputs can be specified, and the model inputs need to be correctly assembled into a tuple. The infer_input_info helper can be used to automatically discover the input names used in the PyTorch model, and to format the inputs correctly for usage with torch.onnx.export.
In the below example we provide the necessary input to run the torchvision mobilenet_v2 model.
The input_names and inputs_as_tuple returned can be directly used in the torch.onnx.export call.
This provides the most benefit when there are multiple inputs to the model, and/or if those inputs involve more complex data types such as dictionaries.
```python
import torch
import torchvision
from onnxruntime import tools
model = torchvision.models.mobilenet_v2(pretrained=True)
model.eval()
input0 = torch.zeros((1, 3, 224, 224), dtype=torch.float32)
input_names, inputs_as_tuple = tools.pytorch_export_helpers.infer_input_info(model, input0)
# input_names and inputs_as_tuple can be directly passed to torch.onnx.export
torch.onnx.export(model, inputs_as_tuple, "model.onnx", input_names=input_names, ...)
```

18
docs/reference/mobile/index.md Normal file
View file

@ -0,0 +1,18 @@
---
title: Mobile
parent: Reference
has_children: true
nav_order: 5
# manual TOC as we don't want some low level things to appear in it, but they need to be children of this page
# for the navigation to work as desired. e.g. doco on scripts mentioned in Model Export Helpers
has_toc: false
---
# ONNX Runtime Mobile Reference
{: .no_toc }
<hr>
#### TABLE OF CONTENTS
* [Model Export Helpers](./helpers.md)

85
docs/reference/mobile/make-dynamic-shape-fixed.md Normal file
View file

@ -0,0 +1,85 @@
---
title: Make dynamic input shape fixed
descriptions:
parent: Mobile
grand_parent: Reference
nav_exclude: true
---
# Making dynamic input shapes fixed
{: .no_toc }
If a model can potentially be used with NNAPI or CoreML as reported by the [model usability checker](./model-usability-checker.md), it may require the input shapes to be made 'fixed'. This is because NNAPI and CoreML do not support dynamic input shapes.
For example, often models have a dynamic batch size so that training is more efficient. In mobile scenarios the batch generally has a size of 1. Making the batch size dimension 'fixed' by setting it to 1 may allow NNAPI and CoreML to run of the model.
The helper can be used to update specific dimensions, or the entire input shape.
## Contents
{: .no_toc }
* TOC placeholder
{:toc}
## Usage
```
python -m onnxruntime.tools.make_dynamic_shape_fixed -h
usage: make_dynamic_shape_fixed.py:make_dynamic_shape_fixed_helper [-h] [--dim_param DIM_PARAM] [--dim_value DIM_VALUE] [--input_name INPUT_NAME] [--input_shape INPUT_SHAPE] input_model output_model
Assign a fixed value to a dim_param or input shape Provide either dim_param and dim_value or input_name and input_shape.
positional arguments:
input_model Provide path to ONNX model to update.
output_model Provide path to write updated ONNX model to.
optional arguments:
-h, --help show this help message and exit
--dim_param DIM_PARAM
Symbolic parameter name. Provider dim_value if specified.
--dim_value DIM_VALUE
Value to replace dim_param with in the model. Must be > 0.
--input_name INPUT_NAME
Model input name to replace shape of. Provider input_shape if specified.
--input_shape INPUT_SHAPE
Shape to use for input_shape. Provide comma separated list for the shape. All values must be > 0. e.g. --input_shape 1,3,256,256
```
To determine the update required by the model, it's generally helpful to view the model in [Netron](https://netron.app/) to inspect the inputs.
## Making a symbolic dimension fixed
Here is an example model, viewed using Netron, with a symbolic dimension called 'batch' for the batch size in 'input:0'.
We will update that to use the fixed value of 1.
![Model with symbolic dimension in input shape](../../../images/model-with-symbolic-input-dim.png)
```
python -m onnxruntime.tools.make_dynamic_shape_fixed --dim_param batch --dim_value 1 model.onnx model.fixed.onnx
```
After replacement you should see that the shape for 'input:0' is now 'fixed' with a value of [1, 36, 36, 3]
![Model with symbolic dimension in input shape replaced with fixed value](../../../images/model-with-symbolic-input-dim-fixed.png)
## Making an input shape fixed
Here is an example model that has unnamed dynamic dimensions for the 'x' input. Netron represents these with '?'.
As there is no name for the dimension, we need to update the shape using the `--input_shape` option.
![Model with dynamic input shape](../../../images/model-with-dynamic-inputs.png)
```
python -m onnxruntime.tools.make_dynamic_shape_fixed --input_name x --input_shape 1,3,960,960 model.onnx model.fixed.onnx
```
After replacement you should see that the shape for 'x' is now 'fixed' with a value of [1, 3, 960, 960]
![Updated model with dynamic input shape now having fixed values](../../../images/model-with-dynamic-inputs-fixed.png)

110
docs/reference/mobile/model-usability-checker.md Normal file
View file

@ -0,0 +1,110 @@
---
title: Model Usability Checker
descriptions: ORT Mobile model usability checker.
parent: Mobile
grand_parent: Reference
nav_exclude: true
---
# Model Usability Checker
{: .no_toc }
The model usability checker analyzes an ONNX model regarding its suitability for usage with ORT Mobile, NNAPI and CoreML.
## Contents
{: .no_toc }
* TOC placeholder
{:toc}
## Usage
```
python -m onnxruntime.tools.check_onnx_model_mobile_usability --help
usage: check_onnx_model_mobile_usability.py [-h] [--config_path CONFIG_PATH] [--log_level {debug,info,warning,error}] model_path
Analyze an ONNX model to determine how well it will work in mobile scenarios, and whether it is likely to be able to use the pre-built ONNX Runtime Mobile Android or iOS package.
positional arguments:
model_path Path to ONNX model to check
optional arguments:
-h, --help show this help message and exit
--config_path CONFIG_PATH
Path to required operators and types configuration used to build the pre-built ORT mobile package. (default:
<onnxruntime package install location>\tools\mobile_helpers\mobile_package.required_operators.config)
--log_level {debug,info,warning,error}
Logging level (default: info)
```
## Use with NNAPI and CoreML
The script will check if the operators in the model are supported by ORT's NNAPI Execution Provider (EP) and CoreML EP. Depending on how many operators are supported, and where they are in the model, it will estimate if using NNAPI or CoreML is likely to be beneficial. It is always recommended to performance test to validate.
Example output from this check looks like:
```
INFO: Checking mobilenet_v1_1.0_224_quant.onnx for usability with ORT Mobile.
INFO: Checking NNAPI
INFO: Model should perform well with NNAPI as is: YES
INFO: Checking CoreML
INFO: Model should perform well with CoreML as is: NO
INFO: Re-run with log level of DEBUG for more details on the NNAPI/CoreML issues.
```
If the model has dynamic input shapes an additional check is made to estimate whether making the shapes of fixed size would help. See [onnxruntime.tools.make_dynamic_shape_fixed](./make-dynamic-shape-fixed.md) for more information.
Example output from this check:
```
INFO: Checking abs_free_dimensions.onnx for usability with ORT Mobile.
INFO: Checking NNAPI
INFO: Model should perform well with NNAPI as is: NO
INFO: Checking if model will perform better if the dynamic shapes are fixed...
INFO: Model should perform well with NNAPI if modified to have fixed input shapes: YES
INFO: Shapes can be altered using python -m onnxruntime.tools.make_dynamic_shape_fixed
```
Setting the log level to `debug` will result in significant amounts of diagnostic output that provides in-depth information on why the recommendations were made.
This includes
- information on individual operators that are supported or unsupported by the NNAPI and CoreML EPs
- information on how many groups (a.k.a. partitions) the supported operators are broken into
- the more groups the worse performance will be as we have to switch between the NPU (Neural Processing Unit) and CPU each time we switch between a supported and unsupported group of nodes
## Use with ORT Mobile Pre-Built package
The ONNX opset and operators used in the model are checked to determine if they are supported by the ORT Mobile pre-built package.
Example output if the model can be used as-is:
```
INFO: Checking if pre-built ORT Mobile package can be used with mobilenet_v1_1.0_224_quant.onnx once model is
converted from ONNX to ORT format using onnxruntime.tools.convert_onnx_models_to_ort...
INFO: Model should work with the pre-built package.
```
If the model uses an old ONNX opset, information will be provided on how to update it.
See [onnxruntime.tools.update_onnx_opset](./helpers.md#onnx-model-opset-updater) for more information.
Example output:
```
INFO: Checking if pre-built ORT Mobile package can be used with abs_free_dimensions.onnx once model is converted
from ONNX to ORT format using onnxruntime.tools.convert_onnx_models_to_ort...
INFO: Model uses ONNX opset 9.
INFO: The pre-built package only supports ONNX opsets [12, 13, 14, 15].
INFO: Please try updating the ONNX model opset to a supported version using
python -m onnxruntime.tools.onnx_model_utils.update_onnx_opset ...
```
## Recommendation
Finally the script will provide information on how to [convert the model to the ORT format](../ort-format-models.md) required by ORT Mobile, and recommend which of the two ORT format models to use.
```
INFO: Run `python -m onnxruntime.tools.convert_onnx_models_to_ort ...` to convert the ONNX model to ORT format.
By default, the conversion tool will create an ORT format model with saved optimizations which can potentially be
applied at runtime (with a .with_runtime_opt.ort file extension) for use with NNAPI or CoreML, and a fully
optimized ORT format model (with a .ort file extension) for use with the CPU EP.
INFO: As NNAPI or CoreML may provide benefits with this model it is recommended to compare the performance of
the <model>.with_runtime_opt.ort model using the NNAPI EP on Android, and the CoreML EP on iOS, against the
performance of the <model>.ort model using the CPU EP.
```

13
docs/performance/ort-format-model-runtime-optimization.md → docs/reference/mobile/ort-format-model-runtime-optimization.md
View file

@ -1,7 +1,8 @@
---
title: ORT Format Model Runtime Optimization
parent: Performance
nav_order: 4
parent: Mobile
grand_parent: Reference
nav_order: 2
---
{::options toc_levels="2" /}
@ -15,15 +16,15 @@ nav_order: 4
## Background
The full ONNX Runtime build supports [graph optimizations](./graph-optimizations.md) at runtime for ONNX models.
The full ONNX Runtime build supports [graph optimizations](../../performance/graph-optimizations.md) at runtime for ONNX models.
The ORT format model was designed to be used with ONNX Runtime [minimal builds](../build/custom.md#minimal-build) for environments where smaller binary size is important. To reduce the binary size, some or all of the graph optimizer code is excluded from a minimal build. As such, ONNX models and ORT format models do not share the same graph optimization process.
The ORT format model was designed to be used with ONNX Runtime [minimal builds](../../build/custom.md#minimal-build) for environments where smaller binary size is important. To reduce the binary size, some or all of the graph optimizer code is excluded from a minimal build. As such, ONNX models and ORT format models do not share the same graph optimization process.
In ONNX Runtime **1.11 and later**, there is limited support for graph optimizations at runtime for ORT format models. This only applies to extended minimal builds or full builds.
In ONNX Runtime **1.10 and earlier**, there is **no support** for graph optimizations at runtime for ORT format models. Any graph optimizations must be done at model conversion time.
As a rule, [basic graph optimizations](./graph-optimizations.md#basic-graph-optimizations) are semantics-preserving and result in a valid ONNX graph. The basic optimizations can and generally should be baked in to the converted ORT format model at conversion time - this is the default behavior of the conversion script. In fact, any runtime optimization support for ORT format models will not include basic optimizations at all.
As a rule, [basic graph optimizations](../../performance/graph-optimizations.md#basic-graph-optimizations) are semantics-preserving and result in a valid ONNX graph. The basic optimizations can and generally should be baked in to the converted ORT format model at conversion time - this is the default behavior of the conversion script. In fact, any runtime optimization support for ORT format models will not include basic optimizations at all.
## Types of runtime optimization
@ -55,4 +56,4 @@ You can compare the performance of:
- A fully optimized model run with only the CPU EP enabled
- A model with saved runtime optimizations run with additional EPs enabled
The [model usability checker](../tutorials/mobile/helpers/model-usability-checker.md) will provide guidance for a particular model.
The [model usability checker](./model-usability-checker.md) will provide guidance for a particular model.

98
docs/reference/operators/ContribOperators.md
View file

@ -5,24 +5,11 @@ grand_parent: Reference
nav_order: 3
---
# Contrib ops
## Contents
{: .no_toc }
* TOC placeholder
{:toc}
The [contrib ops domain](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/contrib_ops) contains ops that are built in to the runtime by default. Only selected operators are added as contrib ops to avoid increasing the binary size of the core runtime package. When possible, [custom operators](./add-custom-op.md) should be used.
## Contrib Op List
The contrib operator schemas are documented in the ONNX Runtime repository.
| Release | Documentation |
|---------|---------------|
| Main | [https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md) |
| 1.14 | [https://github.com/microsoft/onnxruntime/blob/rel-1.14.0/docs/ContribOperators.md](https://github.com/microsoft/onnxruntime/blob/rel-1.14.0/docs/ContribOperators.md)|
| Current master | [https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md) |
| 1.13 | [https://github.com/microsoft/onnxruntime/blob/rel-1.13.1/docs/ContribOperators.md](https://github.com/microsoft/onnxruntime/blob/rel-1.13.1/docs/ContribOperators.md)|
| 1.12 | [https://github.com/microsoft/onnxruntime/blob/rel-1.12.0/docs/ContribOperators.md](https://github.com/microsoft/onnxruntime/blob/rel-1.12.0/docs/ContribOperators.md)|
| 1.11 | [https://github.com/microsoft/onnxruntime/blob/rel-1.11.0/docs/ContribOperators.md](https://github.com/microsoft/onnxruntime/blob/rel-1.11.0/docs/ContribOperators.md)|
@ -31,87 +18,4 @@ The contrib operator schemas are documented in the ONNX Runtime repository.
| 1.8 | [https://github.com/microsoft/onnxruntime/blob/rel-1.8.0/docs/ContribOperators.md](https://github.com/microsoft/onnxruntime/blob/rel-1.8.0/docs/ContribOperators.md)
| 1.7 | [https://github.com/microsoft/onnxruntime/blob/rel-1.7.0/docs/ContribOperators.md](https://github.com/microsoft/onnxruntime/blob/rel-1.7.0/docs/ContribOperators.md)|
## Adding Contrib ops
The custom op's schema and shape inference function should be added in [contrib_defs.cc](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/core/graph/contrib_ops/contrib_defs.cc) using `ONNX_CONTRIB_OPERATOR_SCHEMA`. Example: [Inverse op](https://github.com/microsoft/onnxruntime/pull/3485)
```c++
ONNX_CONTRIB_OPERATOR_SCHEMA(Inverse)
.SetDomain(kMSDomain) // kMSDomain = "com.microsoft"
.SinceVersion(1) // Same version used at op (symbolic) registration
...
```
A new operator should have complete reference implementation tests and shape inference tests.
Reference implementation python tests should be added in
[onnxruntime/test/python/contrib_ops](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/python/contrib_ops).
E.g., [onnx_test_trilu.py](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/python/contrib_ops/onnx_test_trilu.py)
Shape inference C++ tests should be added in
[onnxruntime/test/contrib_ops](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/contrib_ops).
E.g., [trilu_shape_inference_test.cc](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/contrib_ops/trilu_shape_inference_test.cc)
The operator kernel should be implemented using `Compute` function
under contrib namespace in [onnxruntime/contrib_ops/cpu/](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/contrib_ops/cpu/)
for CPU and [onnxruntime/contrib_ops/cuda/](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/contrib_ops/cuda/) for CUDA.
```c++
namespace onnxruntime {
namespace contrib {
class Inverse final : public OpKernel {
public:
explicit Inverse(const OpKernelInfo& info) : OpKernel(info) {}
Status Compute(OpKernelContext* ctx) const override;
private:
...
};
ONNX_OPERATOR_KERNEL_EX(
Inverse,
kMSDomain,
1,
kCpuExecutionProvider,
KernelDefBuilder()
.TypeConstraint("T", BuildKernelDefConstraints<float, double, MLFloat16>()),
Inverse);
Status Inverse::Compute(OpKernelContext* ctx) const {
... // kernel implementation
}
} // namespace contrib
} // namespace onnxruntime
```
The kernel should be registered in [cpu_contrib_kernels.cc](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/contrib_ops/cpu_contrib_kernels.cc) for CPU and [cuda_contrib_kernels.cc](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/contrib_ops/cuda_contrib_kernels.cc) for CUDA.
Now you should be able to build and install ONNX Runtime to start using your custom op.
### Contrib Op Tests
Tests should be added in [onnxruntime/test/contrib_ops/](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/contrib_ops/).
For example:
```c++
namespace onnxruntime {
namespace test {
// Add a comprehensive set of unit tests for custom op kernel implementation
TEST(InverseContribOpTest, two_by_two_float) {
OpTester test("Inverse", 1, kMSDomain); // custom opset version and domain
test.AddInput<float>("X", {2, 2}, {4, 7, 2, 6});
test.AddOutput<float>("Y", {2, 2}, {0.6f, -0.7f, -0.2f, 0.4f});
test.Run();
}
...
} // namespace test
} // namespace onnxruntime
```

3
docs/reference/operators/MobileOps.md
View file

@ -10,9 +10,6 @@ These are the operators and types included in the ORT Mobile pre-built packages
| Release | Documentation |
|---------|---------------|
| 1.14 | [Pre-Built Package Support](./mobile_package_op_type_support_1.14.md)|
| 1.13 | [Pre-Built Package Support](./mobile_package_op_type_support_1.13.md)|
| 1.12 | [Pre-Built Package Support](./mobile_package_op_type_support_1.12.md)|
| 1.11 | [Pre-Built Package Support](./mobile_package_op_type_support_1.11.md)|
| 1.10 | [Pre-Built Package Support](./mobile_package_op_type_support_1.10.md)|
| 1.9 | [Pre-Built Package Support](./mobile_package_op_type_support_1.9.md)|

1
docs/reference/operators/OperatorKernels.md
View file

@ -10,7 +10,6 @@ The operator kernels supported by the CPU Execution Provider and CUDA Execution
| Release | Documentation |
|---------|---------------|
| Current main | [https://github.com/microsoft/onnxruntime/blob/main/docs/OperatorKernels.md](https://github.com/microsoft/onnxruntime/blob/main/docs/OperatorKernels.md) |
| 1.14 | [https://github.com/microsoft/onnxruntime/blob/rel-1.14.0/docs/OperatorKernels.md](https://github.com/microsoft/onnxruntime/blob/rel-1.14.0/docs/OperatorKernels.md)|
| 1.13 | [https://github.com/microsoft/onnxruntime/blob/rel-1.13.1/docs/OperatorKernels.md](https://github.com/microsoft/onnxruntime/blob/rel-1.13.1/docs/OperatorKernels.md)|
| 1.12 | [https://github.com/microsoft/onnxruntime/blob/rel-1.12.0/docs/OperatorKernels.md](https://github.com/microsoft/onnxruntime/blob/rel-1.12.0/docs/OperatorKernels.md)|
| 1.11 | [https://github.com/microsoft/onnxruntime/blob/rel-1.11.0/docs/OperatorKernels.md](https://github.com/microsoft/onnxruntime/blob/rel-1.11.0/docs/OperatorKernels.md)|

96
docs/reference/operators/add-custom-op.md
View file

@ -7,7 +7,7 @@ nav_order: 4
# Custom operators
{: .no_toc }
ONNX Runtime provides options to run custom operators that are not official ONNX operators. Note that custom operators differ from [contrib ops](./ContribOperators.md), which are selected unofficial ONNX operators that are built in directly to ORT.
ONNX Runtime provides options to run custom operators that are not official ONNX operators.
## Contents
{: .no_toc }
@ -94,7 +94,7 @@ Ort::SessionOptions session_options;
session_options.Add(domain); // Add the domain to the session options.
// Create a session.
Ort::Session session(env, ORT_TSTR("my_model_with_custom_ops.onnx"), session_options);
Ort::Session session(env, "my_model_with_custom_ops.onnx", session_options);
```
## Create a library of custom operators
Custom operators can be defined in a separate shared library (e.g., a .dll on Windows or a .so on Linux). A custom operator library must export and implement a `RegisterCustomOps` function. The `RegisterCustomOps` function adds a `Ort::CustomOpDomain` containing the library's custom operators to the provided session options.
@ -183,9 +183,9 @@ Once compiled, the custom operator shared library can then be registered with an
Ort::Env env;
Ort::SessionOptions session_options;
session_options.RegisterCustomOpsLibrary_V2(ORT_TSTR("my_custom_op.dll"));
session_options.RegisterCustomOpsLibrary(L"my_custom_op.dll");
Ort::Session session(env, ORT_TSTR("my_model.onnx"), session_options);
Ort::Session session(env, "my_model.onnx", session_options);
```
### Examples
@ -300,8 +300,94 @@ custom_op_configs.AddConfig("OpenVINO_Wrapper", "device_type", "CPU");
// Register custom op library and pass in the custom op configs (optional).
session_options.RegisterCustomOpsLibrary("MyOpenVINOWrapper_Lib.so", custom_op_configs);
Ort::Session session(env, ORT_TSTR("custom_op_mnist_ov_wrapper.onnx"), session_options);
Ort::Session session(env, "custom_op_mnist_ov_wrapper.onnx", session_options);
```
Refer to the [complete OpenVINO custom operator wrapper example](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/testdata/custom_op_openvino_wrapper_library) for more details. To create an ONNX model that wraps an external model or weights, refer to the [create_custom_op_wrapper.py tool](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/custom_op_wrapper/create_custom_op_wrapper.py).
## Contrib ops
The [contrib ops domain](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/contrib_ops) contains ops that are built in to the runtime by default. However most new operators should not be added here to avoid increasing binary size of the core runtime package.
See for example the Inverse op added in [#3485](https://github.com/microsoft/onnxruntime/pull/3485).
The custom op's schema and shape inference function should be added in [contrib_defs.cc](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/core/graph/contrib_ops/contrib_defs.cc) using `ONNX_CONTRIB_OPERATOR_SCHEMA`.
```c++
ONNX_CONTRIB_OPERATOR_SCHEMA(Inverse)
.SetDomain(kMSDomain) // kMSDomain = "com.microsoft"
.SinceVersion(1) // Same version used at op (symbolic) registration
...
```
A new operator should have complete reference implementation tests and shape inference tests.
Reference implementation python tests should be added in
[onnxruntime/test/python/contrib_ops](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/python/contrib_ops).
E.g., [onnx_test_trilu.py](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/python/contrib_ops/onnx_test_trilu.py)
Shape inference C++ tests should be added in
[onnxruntime/test/contrib_ops](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/contrib_ops).
E.g., [trilu_shape_inference_test.cc](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/contrib_ops/trilu_shape_inference_test.cc)
The operator kernel should be implemented using `Compute` function
under contrib namespace in [onnxruntime/contrib_ops/cpu/](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/contrib_ops/cpu/)
for CPU and [onnxruntime/contrib_ops/cuda/](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/contrib_ops/cuda/) for CUDA.
```c++
namespace onnxruntime {
namespace contrib {
class Inverse final : public OpKernel {
public:
explicit Inverse(const OpKernelInfo& info) : OpKernel(info) {}
Status Compute(OpKernelContext* ctx) const override;
private:
...
};
ONNX_OPERATOR_KERNEL_EX(
Inverse,
kMSDomain,
1,
kCpuExecutionProvider,
KernelDefBuilder()
.TypeConstraint("T", BuildKernelDefConstraints<float, double, MLFloat16>()),
Inverse);
Status Inverse::Compute(OpKernelContext* ctx) const {
... // kernel implementation
}
} // namespace contrib
} // namespace onnxruntime
```
The kernel should be registered in [cpu_contrib_kernels.cc](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/contrib_ops/cpu_contrib_kernels.cc) for CPU and [cuda_contrib_kernels.cc](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/contrib_ops/cuda_contrib_kernels.cc) for CUDA.
Now you should be able to build and install ONNX Runtime to start using your custom op.
### Contrib Op Tests
Tests should be added in [onnxruntime/test/contrib_ops/](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/contrib_ops/).
For example:
```c++
namespace onnxruntime {
namespace test {
// Add a comprehensive set of unit tests for custom op kernel implementation
TEST(InverseContribOpTest, two_by_two_float) {
OpTester test("Inverse", 1, kMSDomain); // custom opset version and domain
test.AddInput<float>("X", {2, 2}, {4, 7, 2, 6});
test.AddOutput<float>("Y", {2, 2}, {0.6f, -0.7f, -0.2f, 0.4f});
test.Run();
}
...
} // namespace test
} // namespace onnxruntime
```

139
docs/reference/operators/mobile_package_op_type_support_1.12.md
View file

@ -1,139 +0,0 @@
---
title: ORT 1.12 Mobile Package Operators
parent: Operators
grand_parent: Reference
nav_exclude: true
---
# ONNX Runtime Mobile Pre-Built Package Operator and Type Support
## Supported operators and types
The supported operators and types are based on what is required to support float32 and quantized versions of popular models. The full list of input models used to determine this list is available [here](https://github.com/microsoft/onnxruntime/blob/master/tools/ci_build/github/android/mobile_package.required_operators.readme.txt)
## Supported data input types
- float
- int8_t
- uint8_t
NOTE: Operators used to manipulate dimensions and indices will support int32 and int64.
## Supported Operators
|Operator|Opsets|
|--------|------|
|**ai.onnx**||
|ai.onnx:Abs|12, 13, 14, 15|
|ai.onnx:Add|12, 13, 14, 15|
|ai.onnx:And|12, 13, 14, 15|
|ai.onnx:ArgMax|12, 13, 14, 15|
|ai.onnx:ArgMin|12, 13, 14, 15|
|ai.onnx:AveragePool|12, 13, 14, 15|
|ai.onnx:Cast|12, 13, 14, 15|
|ai.onnx:Ceil|12, 13, 14, 15|
|ai.onnx:Clip|12, 13, 14, 15|
|ai.onnx:Concat|12, 13, 14, 15|
|ai.onnx:ConstantOfShape|12, 13, 14, 15|
|ai.onnx:Conv|12, 13, 14, 15|
|ai.onnx:ConvTranspose|12, 13, 14, 15|
|ai.onnx:Cos|12, 13, 14, 15|
|ai.onnx:CumSum|12, 13, 14, 15|
|ai.onnx:DepthToSpace|12, 13, 14, 15|
|ai.onnx:DequantizeLinear|12, 13, 14, 15|
|ai.onnx:Div|12, 13, 14, 15|
|ai.onnx:DynamicQuantizeLinear|12, 13, 14, 15|
|ai.onnx:Elu|12, 13, 14, 15|
|ai.onnx:Equal|12, 13, 14, 15|
|ai.onnx:Erf|12, 13, 14, 15|
|ai.onnx:Exp|12, 13, 14, 15|
|ai.onnx:Expand|12, 13, 14, 15|
|ai.onnx:Flatten|12, 13, 14, 15|
|ai.onnx:Floor|12, 13, 14, 15|
|ai.onnx:Gather|12, 13, 14, 15|
|ai.onnx:GatherND|12, 13, 14, 15|
|ai.onnx:Gemm|12, 13, 14, 15|
|ai.onnx:GlobalAveragePool|12, 13, 14, 15|
|ai.onnx:Greater|12, 13, 14, 15|
|ai.onnx:GreaterOrEqual|12, 13, 14, 15|
|ai.onnx:HardSigmoid|12, 13, 14, 15|
|ai.onnx:Identity|12, 13, 14, 15|
|ai.onnx:If|12, 13, 14, 15|
|ai.onnx:InstanceNormalization|12, 13, 14, 15|
|ai.onnx:LRN|12, 13, 14, 15|
|ai.onnx:LayerNormalization|1|
|ai.onnx:LeakyRelu|12, 13, 14, 15|
|ai.onnx:Less|12, 13, 14, 15|
|ai.onnx:LessOrEqual|12, 13, 14, 15|
|ai.onnx:Log|12, 13, 14, 15|
|ai.onnx:LogSoftmax|12, 13, 14, 15|
|ai.onnx:Loop|12, 13, 14, 15|
|ai.onnx:MatMul|12, 13, 14, 15|
|ai.onnx:MatMulInteger|12, 13, 14, 15|
|ai.onnx:Max|12, 13, 14, 15|
|ai.onnx:MaxPool|12, 13, 14, 15|
|ai.onnx:Mean|12, 13, 14, 15|
|ai.onnx:Min|12, 13, 14, 15|
|ai.onnx:Mul|12, 13, 14, 15|
|ai.onnx:Neg|12, 13, 14, 15|
|ai.onnx:NonMaxSuppression|12, 13, 14, 15|
|ai.onnx:NonZero|12, 13, 14, 15|
|ai.onnx:Not|12, 13, 14, 15|
|ai.onnx:Or|12, 13, 14, 15|
|ai.onnx:PRelu|12, 13, 14, 15|
|ai.onnx:Pad|12, 13, 14, 15|
|ai.onnx:Pow|12, 13, 14, 15|
|ai.onnx:QLinearConv|12, 13, 14, 15|
|ai.onnx:QLinearMatMul|12, 13, 14, 15|
|ai.onnx:QuantizeLinear|12, 13, 14, 15|
|ai.onnx:Range|12, 13, 14, 15|
|ai.onnx:Reciprocal|12, 13, 14, 15|
|ai.onnx:ReduceMax|12, 13, 14, 15|
|ai.onnx:ReduceMean|12, 13, 14, 15|
|ai.onnx:ReduceMin|12, 13, 14, 15|
|ai.onnx:ReduceProd|12, 13, 14, 15|
|ai.onnx:ReduceSum|12, 13, 14, 15|
|ai.onnx:Relu|12, 13, 14, 15|
|ai.onnx:Reshape|12, 13, 14, 15|
|ai.onnx:Resize|12, 13, 14, 15|
|ai.onnx:ReverseSequence|12, 13, 14, 15|
|ai.onnx:Round|12, 13, 14, 15|
|ai.onnx:Scan|12, 13, 14, 15|
|ai.onnx:ScatterND|12, 13, 14, 15|
|ai.onnx:Shape|12, 13, 14, 15|
|ai.onnx:Sigmoid|12, 13, 14, 15|
|ai.onnx:Sin|12, 13, 14, 15|
|ai.onnx:Size|12, 13, 14, 15|
|ai.onnx:Slice|12, 13, 14, 15|
|ai.onnx:Softmax|12, 13, 14, 15|
|ai.onnx:SpaceToDepth|12, 13, 14, 15|
|ai.onnx:Split|12, 13, 14, 15|
|ai.onnx:Sqrt|12, 13, 14, 15|
|ai.onnx:Squeeze|12, 13, 14, 15|
|ai.onnx:Sub|12, 13, 14, 15|
|ai.onnx:Sum|12, 13, 14, 15|
|ai.onnx:Tanh|12, 13, 14, 15|
|ai.onnx:ThresholdedRelu|12, 13, 14, 15|
|ai.onnx:Tile|12, 13, 14, 15|
|ai.onnx:TopK|12, 13, 14, 15|
|ai.onnx:Transpose|12, 13, 14, 15|
|ai.onnx:Unique|12, 13, 14, 15|
|ai.onnx:Unsqueeze|12, 13, 14, 15|
|ai.onnx:Where|12, 13, 14, 15|
|||
|**com.microsoft**||
|com.microsoft:DynamicQuantizeMatMul|1|
|com.microsoft:FusedConv|1|
|com.microsoft:FusedGemm|1|
|com.microsoft:FusedMatMul|1|
|com.microsoft:Gelu|1|
|com.microsoft:MatMulIntegerToFloat|1|
|com.microsoft:NhwcMaxPool|1|
|com.microsoft:QLinearAdd|1|
|com.microsoft:QLinearAveragePool|1|
|com.microsoft:QLinearConv|1|
|com.microsoft:QLinearGlobalAveragePool|1|
|com.microsoft:QLinearLeakyRelu|1|
|com.microsoft:QLinearMul|1|
|com.microsoft:QLinearSigmoid|1|
|||

139
docs/reference/operators/mobile_package_op_type_support_1.13.md
View file

@ -1,139 +0,0 @@
---
title: ORT 1.13 Mobile Package Operators
parent: Operators
grand_parent: Reference
nav_exclude: true
---
# ONNX Runtime Mobile Pre-Built Package Operator and Type Support
## Supported operators and types
The supported operators and types are based on what is required to support float32 and quantized versions of popular models. The full list of input models used to determine this list is available [here](https://github.com/microsoft/onnxruntime/blob/main/tools/ci_build/github/android/mobile_package.required_operators.readme.txt)
## Supported data input types
- float
- int8_t
- uint8_t
NOTE: Operators used to manipulate dimensions and indices will support int32 and int64.
## Supported Operators
|Operator|Opsets|
|--------|------|
|**ai.onnx**||
|ai.onnx:Abs|12, 13, 14, 15|
|ai.onnx:Add|12, 13, 14, 15|
|ai.onnx:And|12, 13, 14, 15|
|ai.onnx:ArgMax|12, 13, 14, 15|
|ai.onnx:ArgMin|12, 13, 14, 15|
|ai.onnx:AveragePool|12, 13, 14, 15|
|ai.onnx:Cast|12, 13, 14, 15|
|ai.onnx:Ceil|12, 13, 14, 15|
|ai.onnx:Clip|12, 13, 14, 15|
|ai.onnx:Concat|12, 13, 14, 15|
|ai.onnx:ConstantOfShape|12, 13, 14, 15|
|ai.onnx:Conv|12, 13, 14, 15|
|ai.onnx:ConvTranspose|12, 13, 14, 15|
|ai.onnx:Cos|12, 13, 14, 15|
|ai.onnx:CumSum|12, 13, 14, 15|
|ai.onnx:DepthToSpace|12, 13, 14, 15|
|ai.onnx:DequantizeLinear|12, 13, 14, 15|
|ai.onnx:Div|12, 13, 14, 15|
|ai.onnx:DynamicQuantizeLinear|12, 13, 14, 15|
|ai.onnx:Elu|12, 13, 14, 15|
|ai.onnx:Equal|12, 13, 14, 15|
|ai.onnx:Erf|12, 13, 14, 15|
|ai.onnx:Exp|12, 13, 14, 15|
|ai.onnx:Expand|12, 13, 14, 15|
|ai.onnx:Flatten|12, 13, 14, 15|
|ai.onnx:Floor|12, 13, 14, 15|
|ai.onnx:Gather|12, 13, 14, 15|
|ai.onnx:GatherND|12, 13, 14, 15|
|ai.onnx:Gemm|12, 13, 14, 15|
|ai.onnx:GlobalAveragePool|12, 13, 14, 15|
|ai.onnx:Greater|12, 13, 14, 15|
|ai.onnx:GreaterOrEqual|12, 13, 14, 15|
|ai.onnx:HardSigmoid|12, 13, 14, 15|
|ai.onnx:Identity|12, 13, 14, 15|
|ai.onnx:If|12, 13, 14, 15|
|ai.onnx:InstanceNormalization|12, 13, 14, 15|
|ai.onnx:LRN|12, 13, 14, 15|
|ai.onnx:LayerNormalization|1|
|ai.onnx:LeakyRelu|12, 13, 14, 15|
|ai.onnx:Less|12, 13, 14, 15|
|ai.onnx:LessOrEqual|12, 13, 14, 15|
|ai.onnx:Log|12, 13, 14, 15|
|ai.onnx:LogSoftmax|12, 13, 14, 15|
|ai.onnx:Loop|12, 13, 14, 15|
|ai.onnx:MatMul|12, 13, 14, 15|
|ai.onnx:MatMulInteger|12, 13, 14, 15|
|ai.onnx:Max|12, 13, 14, 15|
|ai.onnx:MaxPool|12, 13, 14, 15|
|ai.onnx:Mean|12, 13, 14, 15|
|ai.onnx:Min|12, 13, 14, 15|
|ai.onnx:Mul|12, 13, 14, 15|
|ai.onnx:Neg|12, 13, 14, 15|
|ai.onnx:NonMaxSuppression|12, 13, 14, 15|
|ai.onnx:NonZero|12, 13, 14, 15|
|ai.onnx:Not|12, 13, 14, 15|
|ai.onnx:Or|12, 13, 14, 15|
|ai.onnx:PRelu|12, 13, 14, 15|
|ai.onnx:Pad|12, 13, 14, 15|
|ai.onnx:Pow|12, 13, 14, 15|
|ai.onnx:QLinearConv|12, 13, 14, 15|
|ai.onnx:QLinearMatMul|12, 13, 14, 15|
|ai.onnx:QuantizeLinear|12, 13, 14, 15|
|ai.onnx:Range|12, 13, 14, 15|
|ai.onnx:Reciprocal|12, 13, 14, 15|
|ai.onnx:ReduceMax|12, 13, 14, 15|
|ai.onnx:ReduceMean|12, 13, 14, 15|
|ai.onnx:ReduceMin|12, 13, 14, 15|
|ai.onnx:ReduceProd|12, 13, 14, 15|
|ai.onnx:ReduceSum|12, 13, 14, 15|
|ai.onnx:Relu|12, 13, 14, 15|
|ai.onnx:Reshape|12, 13, 14, 15|
|ai.onnx:Resize|12, 13, 14, 15|
|ai.onnx:ReverseSequence|12, 13, 14, 15|
|ai.onnx:Round|12, 13, 14, 15|
|ai.onnx:Scan|12, 13, 14, 15|
|ai.onnx:ScatterND|12, 13, 14, 15|
|ai.onnx:Shape|12, 13, 14, 15|
|ai.onnx:Sigmoid|12, 13, 14, 15|
|ai.onnx:Sin|12, 13, 14, 15|
|ai.onnx:Size|12, 13, 14, 15|
|ai.onnx:Slice|12, 13, 14, 15|
|ai.onnx:Softmax|12, 13, 14, 15|
|ai.onnx:SpaceToDepth|12, 13, 14, 15|
|ai.onnx:Split|12, 13, 14, 15|
|ai.onnx:Sqrt|12, 13, 14, 15|
|ai.onnx:Squeeze|12, 13, 14, 15|
|ai.onnx:Sub|12, 13, 14, 15|
|ai.onnx:Sum|12, 13, 14, 15|
|ai.onnx:Tanh|12, 13, 14, 15|
|ai.onnx:ThresholdedRelu|12, 13, 14, 15|
|ai.onnx:Tile|12, 13, 14, 15|
|ai.onnx:TopK|12, 13, 14, 15|
|ai.onnx:Transpose|12, 13, 14, 15|
|ai.onnx:Unique|12, 13, 14, 15|
|ai.onnx:Unsqueeze|12, 13, 14, 15|
|ai.onnx:Where|12, 13, 14, 15|
|||
|**com.microsoft**||
|com.microsoft:DynamicQuantizeMatMul|1|
|com.microsoft:FusedConv|1|
|com.microsoft:FusedGemm|1|
|com.microsoft:FusedMatMul|1|
|com.microsoft:Gelu|1|
|com.microsoft:MatMulIntegerToFloat|1|
|com.microsoft:NhwcMaxPool|1|
|com.microsoft:QLinearAdd|1|
|com.microsoft:QLinearAveragePool|1|
|com.microsoft:QLinearConv|1|
|com.microsoft:QLinearGlobalAveragePool|1|
|com.microsoft:QLinearLeakyRelu|1|
|com.microsoft:QLinearMul|1|
|com.microsoft:QLinearSigmoid|1|
|||

139
docs/reference/operators/mobile_package_op_type_support_1.14.md
View file

@ -1,139 +0,0 @@
---
title: ORT 1.14 Mobile Package Operators
parent: Operators
grand_parent: Reference
nav_exclude: true
---
# ONNX Runtime Mobile Pre-Built Package Operator and Type Support
## Supported operators and types
The supported operators and types are based on what is required to support float32 and quantized versions of popular models. The full list of input models used to determine this list is available [here](https://github.com/microsoft/onnxruntime/blob/main/tools/ci_build/github/android/mobile_package.required_operators.readme.txt)
## Supported data input types
- float
- int8_t
- uint8_t
NOTE: Operators used to manipulate dimensions and indices will support int32 and int64.
## Supported Operators
|Operator|Opsets|
|--------|------|
|**ai.onnx**||
|ai.onnx:Abs|12, 13, 14, 15|
|ai.onnx:Add|12, 13, 14, 15|
|ai.onnx:And|12, 13, 14, 15|
|ai.onnx:ArgMax|12, 13, 14, 15|
|ai.onnx:ArgMin|12, 13, 14, 15|
|ai.onnx:AveragePool|12, 13, 14, 15|
|ai.onnx:Cast|12, 13, 14, 15|
|ai.onnx:Ceil|12, 13, 14, 15|
|ai.onnx:Clip|12, 13, 14, 15|
|ai.onnx:Concat|12, 13, 14, 15|
|ai.onnx:ConstantOfShape|12, 13, 14, 15|
|ai.onnx:Conv|12, 13, 14, 15|
|ai.onnx:ConvTranspose|12, 13, 14, 15|
|ai.onnx:Cos|12, 13, 14, 15|
|ai.onnx:CumSum|12, 13, 14, 15|
|ai.onnx:DepthToSpace|12, 13, 14, 15|
|ai.onnx:DequantizeLinear|12, 13, 14, 15|
|ai.onnx:Div|12, 13, 14, 15|
|ai.onnx:DynamicQuantizeLinear|12, 13, 14, 15|
|ai.onnx:Elu|12, 13, 14, 15|
|ai.onnx:Equal|12, 13, 14, 15|
|ai.onnx:Erf|12, 13, 14, 15|
|ai.onnx:Exp|12, 13, 14, 15|
|ai.onnx:Expand|12, 13, 14, 15|
|ai.onnx:Flatten|12, 13, 14, 15|
|ai.onnx:Floor|12, 13, 14, 15|
|ai.onnx:Gather|12, 13, 14, 15|
|ai.onnx:GatherND|12, 13, 14, 15|
|ai.onnx:Gemm|12, 13, 14, 15|
|ai.onnx:GlobalAveragePool|12, 13, 14, 15|
|ai.onnx:Greater|12, 13, 14, 15|
|ai.onnx:GreaterOrEqual|12, 13, 14, 15|
|ai.onnx:HardSigmoid|12, 13, 14, 15|
|ai.onnx:Identity|12, 13, 14, 15|
|ai.onnx:If|12, 13, 14, 15|
|ai.onnx:InstanceNormalization|12, 13, 14, 15|
|ai.onnx:LRN|12, 13, 14, 15|
|ai.onnx:LayerNormalization|1|
|ai.onnx:LeakyRelu|12, 13, 14, 15|
|ai.onnx:Less|12, 13, 14, 15|
|ai.onnx:LessOrEqual|12, 13, 14, 15|
|ai.onnx:Log|12, 13, 14, 15|
|ai.onnx:LogSoftmax|12, 13, 14, 15|
|ai.onnx:Loop|12, 13, 14, 15|
|ai.onnx:MatMul|12, 13, 14, 15|
|ai.onnx:MatMulInteger|12, 13, 14, 15|
|ai.onnx:Max|12, 13, 14, 15|
|ai.onnx:MaxPool|12, 13, 14, 15|
|ai.onnx:Mean|12, 13, 14, 15|
|ai.onnx:Min|12, 13, 14, 15|
|ai.onnx:Mul|12, 13, 14, 15|
|ai.onnx:Neg|12, 13, 14, 15|
|ai.onnx:NonMaxSuppression|12, 13, 14, 15|
|ai.onnx:NonZero|12, 13, 14, 15|
|ai.onnx:Not|12, 13, 14, 15|
|ai.onnx:Or|12, 13, 14, 15|
|ai.onnx:PRelu|12, 13, 14, 15|
|ai.onnx:Pad|12, 13, 14, 15|
|ai.onnx:Pow|12, 13, 14, 15|
|ai.onnx:QLinearConv|12, 13, 14, 15|
|ai.onnx:QLinearMatMul|12, 13, 14, 15|
|ai.onnx:QuantizeLinear|12, 13, 14, 15|
|ai.onnx:Range|12, 13, 14, 15|
|ai.onnx:Reciprocal|12, 13, 14, 15|
|ai.onnx:ReduceMax|12, 13, 14, 15|
|ai.onnx:ReduceMean|12, 13, 14, 15|
|ai.onnx:ReduceMin|12, 13, 14, 15|
|ai.onnx:ReduceProd|12, 13, 14, 15|
|ai.onnx:ReduceSum|12, 13, 14, 15|
|ai.onnx:Relu|12, 13, 14, 15|
|ai.onnx:Reshape|12, 13, 14, 15|
|ai.onnx:Resize|12, 13, 14, 15|
|ai.onnx:ReverseSequence|12, 13, 14, 15|
|ai.onnx:Round|12, 13, 14, 15|
|ai.onnx:Scan|12, 13, 14, 15|
|ai.onnx:ScatterND|12, 13, 14, 15|
|ai.onnx:Shape|12, 13, 14, 15|
|ai.onnx:Sigmoid|12, 13, 14, 15|
|ai.onnx:Sin|12, 13, 14, 15|
|ai.onnx:Size|12, 13, 14, 15|
|ai.onnx:Slice|12, 13, 14, 15|
|ai.onnx:Softmax|12, 13, 14, 15|
|ai.onnx:SpaceToDepth|12, 13, 14, 15|
|ai.onnx:Split|12, 13, 14, 15|
|ai.onnx:Sqrt|12, 13, 14, 15|
|ai.onnx:Squeeze|12, 13, 14, 15|
|ai.onnx:Sub|12, 13, 14, 15|
|ai.onnx:Sum|12, 13, 14, 15|
|ai.onnx:Tanh|12, 13, 14, 15|
|ai.onnx:ThresholdedRelu|12, 13, 14, 15|
|ai.onnx:Tile|12, 13, 14, 15|
|ai.onnx:TopK|12, 13, 14, 15|
|ai.onnx:Transpose|12, 13, 14, 15|
|ai.onnx:Unique|12, 13, 14, 15|
|ai.onnx:Unsqueeze|12, 13, 14, 15|
|ai.onnx:Where|12, 13, 14, 15|
|||
|**com.microsoft**||
|com.microsoft:DynamicQuantizeMatMul|1|
|com.microsoft:FusedConv|1|
|com.microsoft:FusedGemm|1|
|com.microsoft:FusedMatMul|1|
|com.microsoft:Gelu|1|
|com.microsoft:MatMulIntegerToFloat|1|
|com.microsoft:NhwcMaxPool|1|
|com.microsoft:QLinearAdd|1|
|com.microsoft:QLinearAveragePool|1|
|com.microsoft:QLinearConv|1|
|com.microsoft:QLinearGlobalAveragePool|1|
|com.microsoft:QLinearLeakyRelu|1|
|com.microsoft:QLinearMul|1|
|com.microsoft:QLinearSigmoid|1|
|||

2
docs/reference/ort-format-models.md
View file

@ -153,7 +153,7 @@ optional arguments:
**Since ONNX Runtime 1.11**
Specify whether the converted model will be fully optimized ("Fixed") or have saved runtime optimizations ("Runtime"). Both types of models are produced by default. See [here](./../performance/ort-format-model-runtime-optimization.md) for more information.
Specify whether the converted model will be fully optimized ("Fixed") or have saved runtime optimizations ("Runtime"). Both types of models are produced by default. See [here](./mobile/ort-format-model-runtime-optimization.md) for more information.
This replaces the [optimization level](#optimization-level) option from earlier ONNX Runtime versions.

6
docs/reference/releases-servicing.md
View file

@ -7,11 +7,11 @@ nav_order: 1
# ONNX Runtime releases
The current ONNX Runtime release is [1.14](https://github.com/microsoft/onnxruntime/releases/tag/v1.14.0).
The current ONNX Runtime release is [1.13](https://github.com/microsoft/onnxruntime/releases/tag/v1.13.1).
The next release is ONNX Runtime release 1.15.
The next release is ONNX Runtime release 1.14.
Official releases of ONNX Runtime are managed by the core ONNX Runtime team. A new release is published approximately every quarter, and the upcoming roadmap can be found [here](https://github.com/microsoft/onnxruntime/wiki/Upcoming-Release-Roadmap).
Official releases of ONNX Runtime are managed by the core ONNX Runtime team. A new release is published approximately every quarter.
Releases are versioned according to [Versioning](https://github.com/microsoft/onnxruntime/blob/main/docs/Versioning.md) and release branches are prefixed with "rel-". Patch releases may be published periodically between full releases and have their own release branch.

2
docs/tutorials/accelerate-pytorch/pytorch.md
View file

@ -157,7 +157,7 @@ Then you can run the ONNX model in the environment of your choice. The ONNXRunti
// Allocate ONNXRuntime session
auto memory_info = Ort::MemoryInfo::CreateCpu(OrtDeviceAllocator, OrtMemTypeCPU);
Ort::Env env;
Ort::Session session{env, ORT_TSTR("model.onnx"), Ort::SessionOptions{nullptr}};
Ort::Session session{env, L"model.onnx", Ort::SessionOptions{nullptr}};
// Allocate model inputs: fill in shape and size
std::array<float, ...> input{};

334
docs/tutorials/csharp/stable-diffusion-csharp.md
View file

@ -1,334 +0,0 @@
---
title: Stable Diffusion with C#
description: In this tutorial we will learn how to do inferencing for the popular Stable Diffusion deep learning model in C#.
parent: Inference with C#
grand_parent: Tutorials
has_children: false
nav_order: 1
---
# Inference Stable Diffusion with C# and ONNX Runtime
{: .no_toc }
In this tutorial we will learn how to do inferencing for the popular Stable Diffusion deep learning model in C#. Stable Diffusion models take a text prompt and create an image that represents the text. See the example below:
```text
"make a picture of green tree with flowers around it and a red sky"
```
|----------------------|----------------------|
| <img src="../../../images/sample-output-stablediff.png" width="256" height="256" alt="Image of browser inferencing on sample images."/> | <img src="../../../images/stablediff-example-image.png" width="256" height="256" alt="Image of browser inferencing on sample images."/> |
## Contents
{: .no_toc }
* TOC placeholder
{:toc}
## Prerequisites
This tutorial can be run locally or in the cloud by leveraging Azure Machine Learning compute.
- [Download the Source Code from GitHub](https://github.com/cassiebreviu/StableDiffusion)
To run locally:
- [Visual Studio](https://visualstudio.microsoft.com/downloads/) or [VS Code](https://code.visualstudio.com/Download)
- A GPU enabled machine with CUDA EP Configured. This was built on a GTX 3070 and it has not been tested on anything smaller. Follow [this tutorial to configure CUDA and cuDNN for GPU with ONNX Runtime and C# on Windows 11](https://onnxruntime.ai/docs/tutorials/csharp/csharp-gpu.html)
To run in the cloud with Azure Machine Learning:
- [Azure Subscription](https://azure.microsoft.com/free/)
- [Azure Machine Learning Resource](https://azure.microsoft.com/services/machine-learning/)
## Use Hugging Face to download the Stable Diffusion models
The Hugging Face site has a great library of open source models. We will leverage and download the [ONNX Stable Diffusion models from Hugging Face](https://huggingface.co/models?sort=downloads&search=Stable+Diffusion).
- [Stable Diffusion Models v1.4](https://huggingface.co/CompVis/stable-diffusion-v1-4/tree/onnx)
- [Stable Diffusion Models v1.5](https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/onnx)
Once you have selected a model version repo, click `Files and Versions`, then select the `ONNX` branch. If there isn't an ONNX model branch available, use the `main` branch and convert it to ONNX. See the [ONNX conversion tutorial for PyTorch](https://learn.microsoft.com/windows/ai/windows-ml/tutorials/pytorch-convert-model) for more information.
- Clone the repo:
```text
git lfs install
git clone https://huggingface.co/CompVis/stable-diffusion-v1-4 -b onnx
```
- Copy the folders with the ONNX files to the C# project folder `\StableDiffusion\StableDiffusion`. The folders to copy are: `unet`, `vae_decoder`, `text_encoder`, `safety_checker`.
## Understanding the model in Python with Diffusers from Hugging Face
When taking a prebuilt model and operationalizing it, its useful to take a moment and understand the models in this pipeline. This code is based on the Hugging Face Diffusers Library and Blog. If you want to learn more about how it works [check out this amazing blog post](https://huggingface.co/blog/stable_diffusion) for more details!
## Inference with C#
Now lets start to breakdown how to inference in C#! The `unet` model takes the text embedding of the user prompt created by the [CLIP model](https://huggingface.co/docs/transformers/model_doc/clip) that connects text and image. The latent noisy image is created as a starting point. The scheduler algorithm and the `unet` model work together to denoise the image to create an image that represents the text prompt. Lets look at the code.
## Main Function
The main function sets the prompt, number of inference steps, and the guidance scale. It then calls the `UNet.Inference` function to run the inference.
The properties that need to be set are:
- `prompt` - The text prompt to use for the image
- `num_inference_steps` - The number of steps to run inference for. The more steps the longer it will take to run the inference loop but the image quality should improve.
- `guidance_scale` - The scale for the classifier-free guidance. The higher the number the more it will try to look like the prompt but the image quality may suffer.
- `batch_size` - The number of images to create
- `height` - The height of the image. Default is 512 and must be a multiple of 8.
- `width` - The width of the image. Default is 512 and must be a multiple of 8.
<i>* NOTE: Check out the [Hugging Face Blog](https://huggingface.co/blog/stable_diffusion) for more details.</i>
```csharp
//Default args
var prompt = "make a picture of green tree with flowers around it and a red sky";
// Number of steps
var num_inference_steps = 10;
// Scale for classifier-free guidance
var guidance_scale = 7.5;
//num of images requested
var batch_size = 1;
// Load the tokenizer and text encoder to tokenize and encodethe text.
var textTokenized = TextProcessing.TokenizeText(prompt);
var textPromptEmbeddings = TextProcessing.TextEncode(textTokenized).ToArray();
// Create uncond_input of blank tokens
var uncondInputTokens = TextProcessing.CreateUncondInput();
var uncondEmbedding = TextProcessing.TextEncode(uncondInputTokens).ToArray();
// Concat textEmeddings and uncondEmbedding
DenseTensor<float> textEmbeddings = new DenseTensor<float>(ne[] { 2, 77, 768 });
for (var i = 0; i < textPromptEmbeddings.Length; i++)
{
textEmbeddings[0, i / 768, i % 768] = uncondEmbedding[i];
textEmbeddings[1, i / 768, i % 768] = textPromptEmbeddings[i];
}
var height = 512;
var width = 512;
// Inference Stable Diff
var image = UNet.Inference(num_inference_steps, textEmbeddings,guidance_scale, batch_size, height, width);
// If image failed or was unsafe it will return null.
if( image == null )
{
Console.WriteLine("Unable to create image, please try again.");
}
```
## Tokenization with ONNX Runtime Extensions
The `TextProcessing` class has the functions to tokenize the text prompt and encoded it with the [CLIP model](https://huggingface.co/docs/transformers/model_doc/clip) text encoder.
Instead of reimplementing the CLIP tokenizer in C#, we can leverage the cross-platform CLIP tokenizer implementation in [ONNX Runtime Extensions](https://github.com/microsoft/onnxruntime-extensions). The ONNX Runtime Extensions has a `custom_op_cliptok.onnx` file tokenizer that is used to tokenize the text prompt. The tokenizer is a simple tokenizer that splits the text into words and then converts the words into tokens.
- Text Prompt: a sentence or phrase that represents the image you want to create.
```text
make a picture of green tree with flowers aroundit and a red sky
```
- Text Tokenization: The text prompt is tokenized into a list of tokens. Each token id is a number that represents a word in the sentence, then its filled with a blank token to create the `maxLength` of 77 tokens. The token ids are then converted to a tensor of shape (1,77).
- Below is the code to tokenize the text prompt with ONNX Runtime Extensions.
```csharp
public static int[] TokenizeText(string text)
{
// Create Tokenizer and tokenize the sentence.
var tokenizerOnnxPath = Directory.GetCurrentDirectory().ToString() + ("\\text_tokenizer\\custom_op_cliptok.onnx");
// Create session options for custom op of extensions
var sessionOptions = new SessionOptions();
var customOp = "ortextensions.dll";
sessionOptions.RegisterCustomOpLibraryV2(customOp, out var libraryHandle);
// Create an InferenceSession from the onnx clip tokenizer.
var tokenizeSession = new InferenceSession(tokenizerOnnxPath, sessionOptions);
var inputTensor = new DenseTensor<string>(new string[] { text }, new int[] { 1 });
var inputString = new List<NamedOnnxValue> { NamedOnnxValue.CreateFromTensor<string>("string_input", inputTensor) };
// Run session and send the input data in to get inference output.
var tokens = tokenizeSession.Run(inputString);
var inputIds = (tokens.ToList().First().Value as IEnumerable<long>).ToArray();
Console.WriteLine(String.Join(" ", inputIds));
// Cast inputIds to Int32
var InputIdsInt = inputIds.Select(x => (int)x).ToArray();
var modelMaxLength = 77;
// Pad array with 49407 until length is modelMaxLength
if (InputIdsInt.Length < modelMaxLength)
{
var pad = Enumerable.Repeat(49407, 77 - InputIdsInt.Length).ToArray();
InputIdsInt = InputIdsInt.Concat(pad).ToArray();
}
return InputIdsInt;
}
```
```text
tensor([[49406, 1078, 320, 1674, 539, 1901, 2677, 593, 4023, 1630,
585, 537, 320, 736, 2390, 49407, 49407, 49407, 49407, 49407,
49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407, 49407,
49407, 49407, 49407, 49407, 49407, 49407, 49407]])
```
## Text embedding with the CLIP text encoder model
The tokens are sent to the text encoder model and converted into a tensor of shape (1, 77, 768) where the first dimension is the batch size, the second dimension is the number of tokens and the third dimension is the embedding size. The text encoder is a [OpenAI CLIP](https://openai.com/blog/clip/) model that connects text to images.
The text encoder creates the text embedding which is trained to encode the text prompt into a vector that is used to guide the image generation. The text embedding is then concatenated with the uncond embedding to create the text embeddings that is sent to the unet model for inferencing.
- Text Embedding: A vector of numbers that represents the text prompt created from the tokenization result. The text embedding is created by the `text_encoder` model.
```csharp
public static DenseTensor<float> TextEncoder(int[] tokenizedInput)
{
// Create input tensor.
var input_ids = TensorHelper.CreateTensor(tokenizedInput, new[] { 1, tokenizedInput.Count() });
var input = new List<NamedOnnxValue> { NamedOnnxValue.CreateFromTensor<int>("input_ids", input_ids) };
var textEncoderOnnxPath = Directory.GetCurrentDirectory().ToString() + ("\\text_encoder\\model.onnx");
var encodeSession = new InferenceSession(textEncoderOnnxPath);
// Run inference.
var encoded = encodeSession.Run(input);
var lastHiddenState = (encoded.ToList().First().Value as IEnumerable<float>).ToArray();
var lastHiddenStateTensor = TensorHelper.CreateTensor(lastHiddenState.ToArray(), new[] { 1, 77, 768 });
return lastHiddenStateTensor;
}
```
```text
torch.Size([1, 77, 768])
tensor([[[-0.3884, 0.0229, -0.0522, ..., -0.4899, -0.3066, 0.0675],
[ 0.0520, -0.6046, 1.9268, ..., -0.3985, 0.9645, -0.4424],
[-0.8027, -0.4533, 1.7525, ..., -1.0365, 0.6296, 1.0712],
...,
[-0.6833, 0.3571, -1.1353, ..., -1.4067, 0.0142, 0.3566],
[-0.7049, 0.3517, -1.1524, ..., -1.4381, 0.0090, 0.3777],
[-0.6155, 0.4283, -1.1282, ..., -1.4256, -0.0285, 0.3206]]],
```
## The Inference Loop: UNet model, Timesteps and LMS Scheduler
### Scheduler
The scheduler algorithm and the `unet` model work together to denoise the image to create an image that represents the text prompt. There are different scheduler algorithms that can be used, to learn more about them [check out this blog from Hugging Face](https://huggingface.co/docs/diffusers/using-diffusers/schedulers). In this example we will use the [`LMSDiscreteScheduler`](https://github.com/cassiebreviu/StableDiffusion/blob/main/StableDiffusion/LMSDiscreteScheduler.cs) which was created based on the Hugging Face [scheduling_lms_discrete.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_lms_discrete.py).
### Timesteps
The inference loop is the main loop that runs the scheduler algorithm and the `unet` model. The loop runs for the number of `timesteps` which are calculated by the scheduler algorithm based on the number of inference steps and other parameters.
For this example we have 10 inference steps which calculated the following timesteps:
```csharp
// Get path to model to create inference session.
var modelPath = Directory.GetCurrentDirectory().ToString() + ("\\unet\\model.onnx");
var scheduler = new LMSDiscreteScheduler();
var timesteps = scheduler.SetTimesteps(numInferenceSteps);
```
```text
tensor([999., 888., 777., 666., 555., 444., 333., 222., 111., 0.])
```
### Latents
The `latents` is the noisy image tensor that is used in the model input. It is created using the `GenerateLatentSample` function to create a random tensor of shape (1,4,64,64). The `seed` can be set to a random number or a fixed number. If the `seed` is set to a fixed number the same latent tensor will be used each time. This is useful for debugging or if you want to create the same image each time.
```csharp
var seed = new Random().Next();
var latents = GenerateLatentSample(batchSize, height, width,seed, scheduler.InitNoiseSigma);
```
<img src="../../../images/latents-noise-example.png" width="256" height="256" alt="Image of browser inferencing on sample images."/>
### Inference Loop
For each inference step the latent image is duplicated to create the tensor shape of (2,4,64,64), it is then scaled and inferenced with the unet model. The output tensors (2,4,64,64) are split and guidance is applied. The resulting tensor is then sent into the `LMSDiscreteScheduler` step as part of the denoising process and the resulting tensor from the scheduler step is returned and the loop completes again until the `num_inference_steps` is reached.
```csharp
// Create Inference Session
var unetSession = new InferenceSession(modelPath, options);
var input = new List<NamedOnnxValue>();
for (int t = 0; t < timesteps.Length; t++)
{
// torch.cat([latents] * 2)
var latentModelInput = TensorHelper.Duplicate(latents.ToArray(), new[] { 2, 4, height / 8, width / 8 });
// Scale the input
latentModelInput = scheduler.ScaleInput(latentModelInput, timesteps[t]);
// Create model input of text embeddings, scaled latent image and timestep
input = CreateUnetModelInput(textEmbeddings, latentModelInput, timesteps[t]);
// Run Inference
var output = unetSession.Run(input);
var outputTensor = (output.ToList().First().Value as DenseTensor<float>);
// Split tensors from 2,4,64,64 to 1,4,64,64
var splitTensors = TensorHelper.SplitTensor(outputTensor, new[] { 1, 4, height / 8, width / 8 });
var noisePred = splitTensors.Item1;
var noisePredText = splitTensors.Item2;
// Perform guidance
noisePred = performGuidance(noisePred, noisePredText, guidanceScale);
// LMS Scheduler Step
latents = scheduler.Step(noisePred, timesteps[t], latents);
}
```
## Postprocess the `output` with the VAEDecoder
After the inference loop is complete, the resulting tensor is scaled and then sent to the `vae_decoder` model to decode the image. Lastly the decoded image tensor is converted to an image and saved to disc.
```csharp
public static Tensor<float> Decoder(List<NamedOnnxValue> input)
{
// Load the model which will be used to decode the latents into image space.
var vaeDecoderModelPath = Directory.GetCurrentDirectory().ToString() + ("\\vae_decoder\\model.onnx");
// Create an InferenceSession from the Model Path.
var vaeDecodeSession = new InferenceSession(vaeDecoderModelPath);
// Run session and send the input data in to get inference output.
var output = vaeDecodeSession.Run(input);
var result = (output.ToList().First().Value as Tensor<float>);
return result;
}
public static Image<Rgba32> ConvertToImage(Tensor<float> output, int width = 512, int height = 512, string imageName = "sample")
{
var result = new Image<Rgba32>(width, height);
for (var y = 0; y < height; y++)
{
for (var x = 0; x < width; x++)
{
result[x, y] = new Rgba32(
(byte)(Math.Round(Math.Clamp((output[0, 0, y, x] / 2 + 0.5), 0, 1) * 255)),
(byte)(Math.Round(Math.Clamp((output[0, 1, y, x] / 2 + 0.5), 0, 1) * 255)),
(byte)(Math.Round(Math.Clamp((output[0, 2, y, x] / 2 + 0.5), 0, 1) * 255))
);
}
}
result.Save($@"C:/code/StableDiffusion/{imageName}.png");
return result;
}
```
The result image:
![image](../../../images/sample-output-stablediff.png)
## Conclusion
This is a high level overview of how to run Stable Diffusion in C#. It covered the main concepts and provided examples on how to implement it. To get the full code, check out the [Stable Diffusion C# Sample](https://github.com/cassiebreviu/StableDiffusion).
## Resources
- [Stable Diffusion C# Sample Source Code](https://github.com/cassiebreviu/StableDiffusion)
- [C# API Doc](https://onnxruntime.ai/docs/api/csharp/api)
- [Get Started with C# in ONNX Runtime](https://onnxruntime.ai/docs/get-started/with-csharp.html)
- [Hugging Face Stable Diffusion Blog](https://huggingface.co/blog/stable_diffusion)

8
docs/tutorials/mnist_cpp.md
View file

@ -128,13 +128,13 @@ To make things more interesting, the window painting handler graphs the probabil
### The Ort::Session
1. Creation: The Ort::Session is created inside the MNIST structure here:
```c++
Ort::Session session_{env, ORT_TSTR("model.onnx"), Ort::SessionOptions{nullptr}};
```
Ort::Session session_{env, L"model.onnx", Ort::SessionOptions{nullptr}};
```
[[Source]](https://github.com/microsoft/onnxruntime/blob/521dc757984fbf9770d0051997178fbb9565cd52/samples/c_cxx/MNIST/MNIST.cpp#L43)
2. Setup inputs & outputs: The input & output tensors are created here:
```c++
```
MNIST() {
auto allocator_info = Ort::AllocatorInfo::CreateCpu(OrtDeviceAllocator, OrtMemTypeCPU);
input_tensor_ = Ort::Value::CreateTensor<float>(allocator_info, input_image_.data(), input_image_.size(), input_shape_.data(), input_shape_.size());
@ -146,7 +146,7 @@ To make things more interesting, the window painting handler graphs the probabil
In this usage, we're providing the memory location for the data instead of having Ort allocate the buffers. This is simpler in this case since the buffers are small and can just be fixed members of the MNIST struct.
3. Run: Running the session is done in the Run() method:
```c++
```
int Run() {
const char* input_names[] = {"Input3"};
const char* output_names[] = {"Plus214_Output_0"};

2
docs/tutorials/mobile/helpers/index.md
View file

@ -30,7 +30,7 @@ See [here](./model-usability-checker.md) for more details.
The ORT Mobile pre-built package only supports the most recent ONNX opsets in order to minimize binary size. Most ONNX models can be updated to a newer ONNX opset using this tool. It is recommended to use the latest opset the pre-built package supports, which is currently opset 15.
The ONNX opsets supported by the pre-built package are documented [here](../../../reference/operators/MobileOps.md).
The ONNX opsets supported by the pre-built package are documented [here](../../../reference/operators/mobile_package_op_type_support_1.10.md).
Usage:

5
docs/tutorials/mobile/helpers/make-dynamic-shape-fixed.md
View file

@ -1,10 +1,9 @@
---
title: Make dynamic input shape fixed
descriptions:
parent: ORT Mobile Model Export Helpers
grand_parent: Deploy on Mobile
parent: Helpers
grand_parent: Mobile
nav_order: 2
---
# Making dynamic input shapes fixed

5
docs/tutorials/mobile/helpers/model-usability-checker.md
View file

@ -1,10 +1,9 @@
---
title: Model Usability Checker
descriptions: ORT Mobile model usability checker.
parent: ORT Mobile Model Export Helpers
grand_parent: Deploy on Mobile
parent: Helpers
grand_parent: Mobile
nav_order: 1
---
# Model Usability Checker
{: .no_toc }

2
docs/tutorials/mobile/index.md
View file

@ -13,8 +13,6 @@ ONNX Runtime gives you a variety of options to add machine learning to your mobi
* [Build an objection detection application on iOS](./deploy-ios.md)
* [Build an image classification application on Android](./deploy-android.md)
* [Build an super resolution application on iOS](./superres.md#ios-app)
* [Build an super resolution application on Android](./superres.md#android-app)
## ONNX Runtime mobile application development flow

11
docs/tutorials/mobile/superres.md
View file

@ -33,13 +33,13 @@ We provide a convenient Python script that exports the PyTorch model into ONNX f
```bash
pip install torch
pip install pillow
pip install onnx
pip install onnxruntime
pip install --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/ onnxruntime-extensions
pip install onnxruntime-extensions
pip install pillow
```
A note on versions: the best super resolution results are achieved with ONNX opset 18 (with its support for the Resize operator with anti-aliasing), which is supported by onnx 1.13.0 and onnxruntime 1.14.0 and later. The onnxruntime-extensions package is a pre-release version. The release version will be available soon.
A note on versions: the best super resolution results are achieved with ONNX opset 18 (with its support for the Resize operator with anti-aliasing), which is supported by onnx 1.13.0 and onnxruntime 1.14.0 and later.
2. Then download the script and test image from the onnxruntime-extensions GitHub repository (if you have not already cloned this repository):
@ -102,7 +102,8 @@ Add the following dependencies to the app `build.gradle`:
```gradle
implementation 'com.microsoft.onnxruntime:onnxruntime-android:latest.release'
implementation 'com.microsoft.onnxruntime:onnxruntime-extensions-android:latest.release'
// TODO: update with released version aar package when available
implementation files('libs/onnxruntime-extensions-android-0.6.0.aar')
```
#### Project resources

8
docs/tutorials/web/index.md
View file

@ -13,7 +13,7 @@ You can also integrate machine learning into the server side of your web applica
To see an example of the web development flow in practice, you can follow the steps in the following tutorial to [build a web application to classify images using Next.js](classify-images-nextjs-github-template.md).
For more detail on the steps below, see the [build a web application](./build-web-app.md) with ONNX Runtime reference guide.
For more detail on the steps below, see the [build a web application](../../reference/build-web-app.md) with ONNX Runtime reference guide.
## ONNX Runtime web application development flow
@ -66,11 +66,11 @@ For more detail on the steps below, see the [build a web application](./build-we
Bootstrap your web application according in your web framework of choice e.g. vuejs, reactjs, angularjs.
1. [Add the ONNX Runtime dependency](./build-web-app.md#add-onnx-runtime-web-as-dependency)
1. [Add the ONNX Runtime dependency](../../reference/build-web-app.md#add-onnx-runtime-web-as-dependency)
1. [Consume the onnxruntime-web API in your application](./build-web-app.md#consume-onnxruntime-web-in-your-code)
1. [Consume the onnxruntime-web API in your application](../../reference/build-web-app.md#consume-onnxruntime-web-in-your-code)
1. [Add pre and post processing](./build-web-app.md#pre-and-post-processing) appropriate to your application and model
1. [Add pre and post processing](../../reference/build-web-app.md#pre-and-post-processing) appropriate to your application and model
4. How do I optimize my application?

BIN
images/latents-noise-example.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 5.9 KiB

BIN
images/sample-output-stablediff.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 550 KiB

BIN
images/stablediff-example-image.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 576 KiB

22
index.html
View file

@ -482,8 +482,8 @@
allowfullscreen></iframe>
</div>
<div class="col-12 col-md-4 mb-4 mb-md-0">
<iframe class="video" src="https://www.youtube.com/embed/waIeC3OIn70"
title="v1.14 ONNX Runtime - Release Review" frameborder="0"
<iframe class="video" src="https://www.youtube.com/embed/imjqRdsm2Qw"
title="C++ and OnnxRuntime" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen></iframe>
</div>
@ -492,14 +492,8 @@
<div class="carousel-item">
<div class="row blue-title-columns">
<div class="col-12 col-md-4 mb-4 mb-md-0">
<iframe class="video" src="https://www.youtube.com/embed/imjqRdsm2Qw"
title="C++ and OnnxRuntime" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen></iframe>
</div>
<div class="col-12 col-md-4 mb-4 mb-md-0">
<iframe class="video" src="https://www.youtube.com/embed/2dbXFQxsIDY"
title="ONNX Runtime Azure EP for Hybrid Inferencing on Edge and Cloud" frameborder="0"
<iframe class="video" src="https://www.youtube.com/embed/vo9vlR-TRK4"
title="v1.13 Release Review" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen></iframe>
</div>
@ -509,16 +503,16 @@
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen></iframe>
</div>
</div>
</div>
<div class="carousel-item">
<div class="row blue-title-columns">
<div class="col-12 col-md-4 mb-4 mb-md-0">
<iframe class="video" src="https://www.youtube.com/embed/Z8SX5oXF7dM"
title="Computer vision C# with ONNX Runtime!" frameborder="0"
allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
allowfullscreen></iframe>
</div>
</div>
</div>
<div class="carousel-item">
<div class="row blue-title-columns">
<div class="col-12 col-md-4 mb-4 mb-md-0">
<iframe class="video" src="https://www.youtube.com/embed/W_lUGPMW_Eg"
title="Deploy Transformer Models in the Browser with #ONNXRuntime"