mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-26 19:52:38 +00:00
Update link format and example sections in readme (#1729)
* Fix broken link and minor wording updates * Update links to use relative paths * Update sample section organization * Fix a few more links * Update links to relative paths * Fix link urls * Update links to relative paths * Update link to perf test doc page * Update links to relative paths * Update to relative paths for links * Update link
This commit is contained in:
Faith Xu
GitHub
parent
a0ba25f98f
commit
a60283845b
6 changed files with 68 additions and 55 deletions
74
README.md
74
README.md
|
|
@ -35,15 +35,15 @@
|
|||
***
|
||||
# Key Features
|
||||
## Run any ONNX model
|
||||
ONNX Runtime provides comprehensive support of the ONNX spec and can be used to run all models based on ONNX v1.2.1 and higher. See version compatibility details [here](https://github.com/microsoft/onnxruntime/blob/master/docs/Versioning.md).
|
||||
ONNX Runtime provides comprehensive support of the ONNX spec and can be used to run all models based on ONNX v1.2.1 and higher. See version compatibility details [here](./docs/Versioning.md).
|
||||
|
||||
**Traditional ML support**
|
||||
|
||||
In addition to DNN models, ONNX Runtime fully supports the [ONNX-ML profile](https://github.com/onnx/onnx/blob/master/docs/Operators-ml.md) of the ONNX spec for traditional ML scenarios.
|
||||
|
||||
For the full set of operators and types supported, please see [operator documentation](https://github.com/microsoft/onnxruntime/blob/master/docs/OperatorKernels.md)
|
||||
For the full set of operators and types supported, please see [operator documentation](./docs/OperatorKernels.md)
|
||||
|
||||
*Note: Some operators not supported in the current ONNX version may be available as a [Contrib Operator](https://github.com/microsoft/onnxruntime/blob/master/docs/ContribOperators.md)*
|
||||
*Note: Some operators not supported in the current ONNX version may be available as a [Contrib Operator](./docs/ContribOperators.md)*
|
||||
|
||||
|
||||
## High Performance
|
||||
|
|
@ -51,29 +51,29 @@ ONNX Runtime supports both CPU and GPU. Using various graph optimizations and ac
|
|||
|
||||
Currently ONNX Runtime supports the following accelerators:
|
||||
* MLAS (Microsoft Linear Algebra Subprograms)
|
||||
* [MKL-DNN](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/MKL-DNN-ExecutionProvider.md) - [subgraph optimization](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/MKL-DNN-Subgraphs.md)
|
||||
* [MKL-DNN](./docs/execution_providers/MKL-DNN-ExecutionProvider.md) - [subgraph optimization](./docs/execution_providers/MKL-DNN-Subgraphs.md)
|
||||
* MKL-ML
|
||||
* [Intel nGraph](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/nGraph-ExecutionProvider.md)
|
||||
* [Intel nGraph](./docs/execution_providers/nGraph-ExecutionProvider.md)
|
||||
* CUDA
|
||||
* [TensorRT](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/TensorRT-ExecutionProvider.md)
|
||||
* [OpenVINO](https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/OpenVINO-ExecutionProvider.md)
|
||||
* [Nuphar](docs/execution_providers/Nuphar-ExecutionProvider.md)
|
||||
* [TensorRT](./docs/execution_providers/TensorRT-ExecutionProvider.md)
|
||||
* [OpenVINO](./docs/execution_providers/OpenVINO-ExecutionProvider.md)
|
||||
* [Nuphar](./docs/execution_providers/Nuphar-ExecutionProvider.md)
|
||||
|
||||
Not all variations are supported in the [official release builds](#apis-and-official-builds), but can be built from source following [these instructions](https://github.com/Microsoft/onnxruntime/blob/master/BUILD.md). Find Dockerfiles [here](https://github.com/microsoft/onnxruntime/tree/master/dockerfiles).
|
||||
Not all variations are supported in the [official release builds](#apis-and-official-builds), but can be built from source following [these instructions](./BUILD.md). Find Dockerfiles [here](./dockerfiles).
|
||||
|
||||
We are continuously working to integrate new execution providers for further improvements in latency and efficiency. If you are interested in contributing a new execution provider, please see [this page](docs/AddingExecutionProvider.md).
|
||||
|
||||
|
||||
## Cross Platform
|
||||
[API documentation and package installation](https://github.com/microsoft/onnxruntime#installation)
|
||||
[API documentation and package installation](#installation)
|
||||
|
||||
ONNX Runtime is available for Linux, Windows, Mac with Python, C#, and C APIs, with more to come!
|
||||
If you have specific scenarios that are not currently supported, please share your suggestions and scenario details via [Github Issues](https://github.com/microsoft/onnxruntime/issues).
|
||||
ONNX Runtime is currently available for Linux, Windows, and Mac with Python, C#, C++, and C APIs.
|
||||
If you have specific scenarios that are not supported, please share your suggestions and scenario details via [Github Issues](https://github.com/microsoft/onnxruntime/issues).
|
||||
***
|
||||
# Installation
|
||||
**Quick Start:** The [ONNX-Ecosystem Docker container image](https://github.com/onnx/onnx-docker/tree/master/onnx-ecosystem) is available on Dockerhub and includes ONNX Runtime (CPU, Python), dependencies, tools to convert from various frameworks, and Jupyter notebooks to help get started.
|
||||
|
||||
Additional dockerfiles for some features can be found [here](https://github.com/microsoft/onnxruntime/tree/master/dockerfiles).
|
||||
Additional dockerfiles for some features can be found [here](./dockerfiles).
|
||||
|
||||
## APIs and Official Builds
|
||||
|
||||
|
|
@ -81,7 +81,7 @@ Additional dockerfiles for some features can be found [here](https://github.com/
|
|||
* [Python](https://aka.ms/onnxruntime-python)
|
||||
* [C](docs/C_API.md)
|
||||
* [C#](docs/CSharp_API.md)
|
||||
* [C++](https://github.com/microsoft/onnxruntime/blob/master/include/onnxruntime/core/session/onnxruntime_cxx_api.h)
|
||||
* [C++](./include/onnxruntime/core/session/onnxruntime_cxx_api.h)
|
||||
|
||||
### Official Builds
|
||||
| | CPU (MLAS+Eigen) | CPU (MKL-ML) | GPU (CUDA)
|
||||
|
|
@ -100,7 +100,7 @@ system.
|
|||
* Version: **CUDA 10.0** and **cuDNN 7.3**
|
||||
* Linux Python packages require **CUDA 10.1** and **cuDNN 7.6**
|
||||
* Older ONNX Runtime releases: used **CUDA 9.1** and **cuDNN 7.1** - please refer to [prior release notes](https://github.com/microsoft/onnxruntime/releases) for more details.
|
||||
* Python binaries are compatible with **Python 3.5-3.7**. See [Python Dev Notes](https://github.com/microsoft/onnxruntime/blob/master/docs/Python_Dev_Notes.md). If using `pip` to be download the Python binaries, run `pip install --upgrade pip` prior to downloading.
|
||||
* Python binaries are compatible with **Python 3.5-3.7**. See [Python Dev Notes](./docs/Python_Dev_Notes.md). If using `pip` to be download the Python binaries, run `pip install --upgrade pip` prior to downloading.
|
||||
* Certain operators makes use of system locales. Installation of the **English language package** and configuring `en_US.UTF-8 locale` is required.
|
||||
* For Ubuntu install [language-pack-en package](https://packages.ubuntu.com/search?keywords=language-pack-en)
|
||||
* Run the following commands:
|
||||
|
|
@ -111,7 +111,7 @@ system.
|
|||
## Building from Source
|
||||
If additional build flavors are needed, please find instructions on building from source at [Build ONNX Runtime](BUILD.md). For production scenarios, it's strongly recommended to build from an [official release branch](https://github.com/microsoft/onnxruntime/releases).
|
||||
|
||||
Dockerfiles are available [here](https://github.com/microsoft/onnxruntime/tree/faxu-doc-updates/tools/ci_build/github/linux/docker) to help you get started.
|
||||
Dockerfiles are available [here](./tools/ci_build/github/linux/docker) to help you get started.
|
||||
|
||||
***
|
||||
# Usage
|
||||
|
|
@ -127,24 +127,25 @@ Dockerfiles are available [here](https://github.com/microsoft/onnxruntime/tree/f
|
|||
## Deploying ONNX Runtime
|
||||
ONNX Runtime can be deployed to the cloud for model inferencing using [Azure Machine Learning Services](https://azure.microsoft.com/en-us/services/machine-learning-service). See [detailed instructions](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-build-deploy-onnx) and [sample notebooks](https://github.com/Azure/MachineLearningNotebooks/tree/master/how-to-use-azureml/deployment/onnx).
|
||||
|
||||
**ONNX Runtime Server (beta)** is a hosted application for serving ONNX models using ONNX Runtime, providing a REST API for prediction. Usage details can be found [here](https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Server_Usage.md), and image installation instructions are [here](https://github.com/microsoft/onnxruntime/tree/master/dockerfiles#onnx-runtime-server-preview).
|
||||
**ONNX Runtime Server (beta)** is a hosted application for serving ONNX models using ONNX Runtime, providing a REST API for prediction. Usage details can be found [here](./docs/ONNX_Runtime_Server_Usage.md), and image installation instructions are [here](./dockerfiles#onnx-runtime-server-preview).
|
||||
|
||||
## Performance Tuning
|
||||
ONNX Runtime is open and extensible, supporting a broad set of configurations and execution providers for model acceleration. For performance tuning guidance, please see [this page](https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Perf_Tuning.md).
|
||||
ONNX Runtime is open and extensible, supporting a broad set of configurations and execution providers for model acceleration. For performance tuning guidance, please see [this page](./docs/ONNX_Runtime_Perf_Tuning.md).
|
||||
|
||||
***
|
||||
# Examples and Tutorials
|
||||
## Python
|
||||
* [Basic Inferencing Sample](https://github.com/onnx/onnx-docker/blob/master/onnx-ecosystem/inference_demos/simple_onnxruntime_inference.ipynb)
|
||||
* [Inferencing (Resnet50)](https://github.com/onnx/onnx-docker/blob/master/onnx-ecosystem/inference_demos/resnet50_modelzoo_onnxruntime_inference.ipynb)
|
||||
* [Inferencing samples](https://github.com/onnx/onnx-docker/tree/master/onnx-ecosystem/inference_demos) using [ONNX-Ecosystem Docker image](https://github.com/onnx/onnx-docker/tree/master/onnx-ecosystem)
|
||||
* [Train, Convert, and Inference a SKL pipeline](https://microsoft.github.io/onnxruntime/auto_examples/plot_train_convert_predict.html#sphx-glr-auto-examples-plot-train-convert-predict-py)
|
||||
* [Convert and Inference a Keras model](https://microsoft.github.io/onnxruntime/auto_examples/plot_dl_keras.html#sphx-glr-auto-examples-plot-dl-keras-py)
|
||||
* [ONNX Runtime Server: SSD Single Shot MultiBox Detector](https://github.com/onnx/tutorials/blob/master/tutorials/OnnxRuntimeServerSSDModel.ipynb)
|
||||
* [Running ONNX model tests](https://github.com/microsoft/onnxruntime/blob/master/docs/Model_Test.md)
|
||||
**Inference only**
|
||||
* [Model Inferencing (single node Sigmoid)](https://github.com/onnx/onnx-docker/blob/master/onnx-ecosystem/inference_demos/simple_onnxruntime_inference.ipynb)
|
||||
* [Model Inferencing (Resnet50)](https://github.com/onnx/onnx-docker/blob/master/onnx-ecosystem/inference_demos/resnet50_modelzoo_onnxruntime_inference.ipynb)
|
||||
* [Model Inferencing](https://github.com/onnx/onnx-docker/tree/master/onnx-ecosystem/inference_demos) using [ONNX-Ecosystem Docker image](https://github.com/onnx/onnx-docker/tree/master/onnx-ecosystem)
|
||||
* [Model Inferencing using ONNX Runtime Server (SSD Single Shot MultiBox Detector)](https://github.com/onnx/tutorials/blob/master/tutorials/OnnxRuntimeServerSSDModel.ipynb)
|
||||
|
||||
**Inference with model conversion**
|
||||
* [SKL Pipeline: Train, Convert, and Inference](https://microsoft.github.io/onnxruntime/auto_examples/plot_train_convert_predict.html#sphx-glr-auto-examples-plot-train-convert-predict-py)
|
||||
* [Keras: Convert and Inference](https://microsoft.github.io/onnxruntime/auto_examples/plot_dl_keras.html#sphx-glr-auto-examples-plot-dl-keras-py)
|
||||
|
||||
**Deployment with AzureML**
|
||||
**Inference and deploy through AzureML**
|
||||
* Inferencing using [ONNX Model Zoo](https://github.com/onnx/models) models:
|
||||
* [Facial Expression Recognition](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-inference-facial-expression-recognition-deploy.ipynb)
|
||||
* [MNIST Handwritten Digits](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-inference-mnist-deploy.ipynb)
|
||||
|
|
@ -153,19 +154,26 @@ ONNX Runtime is open and extensible, supporting a broad set of configurations an
|
|||
* [TinyYolo](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-convert-aml-deploy-tinyyolo.ipynb)
|
||||
* Train a model with PyTorch and Inferencing:
|
||||
* [MNIST](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-train-pytorch-aml-deploy-mnist.ipynb)
|
||||
|
||||
* Inferencing with TensorRT Execution Provider on GPU (AKS)
|
||||
* [FER+](https://github.com/microsoft/onnxruntime/blob/master/docs/python/notebooks/onnx-inference-byoc-gpu-cpu-aks.ipynb)
|
||||
|
||||
* GPU: Inferencing with TensorRT Execution Provider (AKS)
|
||||
* [FER+](./docs/python/notebooks/onnx-inference-byoc-gpu-cpu-aks.ipynb)
|
||||
|
||||
**Inference and Deploy wtih Azure IoT Edge**
|
||||
* [Intel OpenVINO](http://aka.ms/onnxruntime-openvino)
|
||||
* [NVIDIA TensorRT on Jetson Nano (ARM64)](http://aka.ms/onnxruntime-arm64)
|
||||
|
||||
**Other**
|
||||
* [Running ONNX model tests](./docs/Model_Test.md)
|
||||
|
||||
|
||||
## C#
|
||||
* [Inferencing Tutorial](https://github.com/microsoft/onnxruntime/blob/master/docs/CSharp_API.md#getting-started)
|
||||
* [Inferencing Tutorial](./docs/CSharp_API.md#getting-started)
|
||||
|
||||
|
||||
## C/C++
|
||||
* [Basic Inferencing (SqueezeNet) - C](https://github.com/microsoft/onnxruntime/blob/master/csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/C_Api_Sample.cpp)
|
||||
* [Basic Inferencing (SqueezeNet) - C++](https://github.com/microsoft/onnxruntime/blob/master/csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/CXX_Api_Sample.cpp)
|
||||
* [Inferencing (MNIST) - C++](https://github.com/microsoft/onnxruntime/tree/master/samples/c_cxx/MNIST)
|
||||
* [C - Inferencing (SqueezeNet)](./csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/C_Api_Sample.cpp)
|
||||
* [C++ - Inferencing (SqueezeNet)](./csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/CXX_Api_Sample.cpp)
|
||||
* [C++ - Inferencing (MNIST)](./samples/c_cxx/MNIST)
|
||||
|
||||
***
|
||||
# Technical Design Details
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ You can load your input data into Tensor<T> objects in several ways. A simple ex
|
|||
int[] dimensions; // and the dimensions of the input is stored here
|
||||
Tensor<float> t1 = new DenseTensor<float>(sourceData, dimensions);
|
||||
|
||||
Here is a [complete sample code](https://github.com/Microsoft/onnxruntime/tree/master/csharp/sample/Microsoft.ML.OnnxRuntime.InferenceSample) that runs inference on a pretrained model.
|
||||
Here is a [complete sample code](../csharp/sample/Microsoft.ML.OnnxRuntime.InferenceSample) that runs inference on a pretrained model.
|
||||
|
||||
## Running on GPU (Optional)
|
||||
If using the GPU package, simply use the appropriate SessionOptions when creating an InferenceSession.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
<h1><span style="color:red">Note: ONNX Runtime Server is still in beta state. It's currently not ready for production environments.</span></h1>
|
||||
<h1><span style="color:red">Note: ONNX Runtime Server is still in beta state and may not be ready for production environments.</span></h1>
|
||||
|
||||
# How to Use ONNX Runtime Server for Prediction
|
||||
|
||||
|
|
@ -43,7 +43,7 @@ http://<your_ip_address>:<port>/v1/models/<your-model-name>/versions/<your-versi
|
|||
|
||||
### Request and Response Payload
|
||||
|
||||
The request and response need to be a protobuf message. The Protobuf definition can be found [here](https://github.com/Microsoft/onnxruntime/blob/master/onnxruntime/server/protobuf/predict.proto).
|
||||
The request and response need to be a protobuf message. The Protobuf definition can be found [here](../onnxruntime/server/protobuf/predict.proto).
|
||||
|
||||
A protobuf message could have two formats: binary and JSON. Usually the binary payload has better latency, in the meanwhile the JSON format is easy for human readability.
|
||||
|
||||
|
|
@ -74,7 +74,7 @@ A simple Jupyter notebook demonstrating the usage of ONNX Runtime server to host
|
|||
|
||||
## GRPC Endpoint
|
||||
|
||||
If you prefer using the GRPC endpoint, the protobuf could be found [here](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/server/protobuf/prediction_service.proto). You could generate your client and make a GRPC call to it. To learn more about how to generate the client code and call to the server, please refer to [the tutorials of GRPC](https://grpc.io/docs/tutorials/).
|
||||
If you prefer using the GRPC endpoint, the protobuf could be found [here](../onnxruntime/server/protobuf/prediction_service.proto). You could generate your client and make a GRPC call to it. To learn more about how to generate the client code and call to the server, please refer to [the tutorials of GRPC](https://grpc.io/docs/tutorials/).
|
||||
|
||||
## Advanced Topics
|
||||
|
||||
|
|
@ -91,7 +91,7 @@ For easy tracking of requests, we provide the following header fields:
|
|||
|
||||
### rsyslog Support
|
||||
|
||||
If you prefer using an ONNX Runtime Server with [rsyslog](https://www.rsyslog.com/) support([build instruction](https://github.com/microsoft/onnxruntime/blob/master/BUILD.md#build-onnx-runtime-server-on-linux)), you should be able to see the log in `/var/log/syslog` after the ONNX Runtime Server runs. For detail about how to use rsyslog, please reference [here](https://www.rsyslog.com/category/guides-for-rsyslog/).
|
||||
If you prefer using an ONNX Runtime Server with [rsyslog](https://www.rsyslog.com/) support([build instruction](../BUILD.md#build-onnx-runtime-server-on-linux)), you should be able to see the log in `/var/log/syslog` after the ONNX Runtime Server runs. For detail about how to use rsyslog, please reference [here](https://www.rsyslog.com/category/guides-for-rsyslog/).
|
||||
|
||||
## Report Issues
|
||||
|
||||
|
|
|
|||
|
|
@ -26,10 +26,10 @@ InferenceSession session_object{so};
|
|||
session_object.RegisterExecutionProvider(std::make_unique<::onnxruntime:: MKLDNNExecutionProvider >());
|
||||
status = session_object.Load(model_file_name);
|
||||
```
|
||||
The C API details are [here](https://github.com/Microsoft/onnxruntime/blob/master/docs/C_API.md#c-api).
|
||||
The C API details are [here](../C_API.md#c-api).
|
||||
|
||||
## Python
|
||||
When using the python wheel from the ONNX Runtime built with MKL-DNN execution provider, it will be automatically prioritized over the CPU execution provider. Python APIs details are [here](https://github.com/Microsoft/onnxruntime/blob/master/docs/python/api_summary.rst#api-summary).
|
||||
When using the python wheel from the ONNX Runtime built with MKL-DNN execution provider, it will be automatically prioritized over the CPU execution provider. Python APIs details are [here](https://aka.ms/onnxruntime-python).
|
||||
|
||||
## Using onnxruntime_perf_test and onnx_test_runner
|
||||
You can test the performance of your ONNX Model with the MKL-DNN execution provider. Use the flag -e mkldnn in [onnxruntime_perf_test](https://github.com/Microsoft/onnxruntime/tree/master/onnxruntime/test/perftest#onnxruntime-performance-test) and [onnx_test_runner](https://github.com/Microsoft/onnxruntime/tree/master/onnxruntime/test/onnx/README.txt)..
|
||||
## Performance Tuning
|
||||
For performance tuning, please see guidance on this page: [ONNX Runtime Perf Tuning](../ONNX_Runtime_Perf_Tuning.md)
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ The TensorRT execution provider in the ONNX Runtime makes use of NVIDIA's [Tenso
|
|||
With the TensorRT execution provider, the ONNX Runtime delivers better inferencing performance on the same hardware compared to generic GPU acceleration.
|
||||
|
||||
### Build TensorRT execution provider
|
||||
Developers can now tap into the power of TensorRT through ONNX Runtime to accelerate inferencing of ONNX models. Instructions to build the TensorRT execution provider from source are available [here](https://github.com/Microsoft/onnxruntime/blob/master/BUILD.md#build). [Dockerfiles](https://github.com/microsoft/onnxruntime/tree/master/dockerfiles#tensorrt-version-preview) are available for convenience.
|
||||
Developers can now tap into the power of TensorRT through ONNX Runtime to accelerate inferencing of ONNX models. Instructions to build the TensorRT execution provider from source are available [here](../../BUILD.md#build). [Dockerfiles](../../dockerfiles#tensorrt-version-preview) are available for convenience.
|
||||
|
||||
### Using the TensorRT execution provider
|
||||
#### C/C++
|
||||
|
|
@ -15,26 +15,28 @@ InferenceSession session_object{so};
|
|||
session_object.RegisterExecutionProvider(std::make_unique<::onnxruntime::TensorrtExecutionProvider>());
|
||||
status = session_object.Load(model_file_name);
|
||||
```
|
||||
The C API details are [here](https://github.com/Microsoft/onnxruntime/blob/master/docs/C_API.md#c-api).
|
||||
The C API details are [here](../C_API.md#c-api).
|
||||
|
||||
### Python
|
||||
When using the Python wheel from the ONNX Runtime build with TensorRT execution provider, it will be automatically prioritized over the default GPU or CPU execution providers. There is no need to separately register the execution provider. Python APIs details are [here](https://microsoft.github.io/onnxruntime/api_summary.html).
|
||||
|
||||
### Performance Tuning
|
||||
To test the performance of your ONNX Model with the TensorRT execution provider, use the flag `-e tensorrt` in [onnxruntime_perf_test](https://github.com/Microsoft/onnxruntime/tree/master/onnxruntime/test/perftest#onnxruntime-performance-test).
|
||||
|
||||
### Sample
|
||||
Please see [this Notebook](https://github.com/microsoft/onnxruntime/blob/master/docs/python/notebooks/onnx-inference-byoc-gpu-cpu-aks.ipynb) for an example of running a model on GPU using ONNX Runtime through Azure Machine Learning Services.
|
||||
Please see [this Notebook](../python/notebooks/onnx-inference-byoc-gpu-cpu-aks.ipynb) for an example of running a model on GPU using ONNX Runtime through Azure Machine Learning Services.
|
||||
|
||||
### Using onnxruntime_perf_test
|
||||
You can test the performance for your ONNX Model with the TensorRT execution provider. Use the flag `-e tensorrt` in [onnxruntime_perf_test](https://github.com/Microsoft/onnxruntime/tree/master/onnxruntime/test/perftest#onnxruntime-performance-test).
|
||||
### Performance Tuning
|
||||
For performance tuning, please see guidance on this page: [ONNX Runtime Perf Tuning](../ONNX_Runtime_Perf_Tuning.md)
|
||||
|
||||
When/if using [onnxruntime_perf_test](../../onnxruntime/test/perftest#onnxruntime-performance-test), use the flag `-e tensorrt`
|
||||
|
||||
### Configuring Engine Max Batch Size and Workspace Size
|
||||
By default TensorRT execution provider builds an ICudaEngine with max batch size = 1 and max workspace size = 1 GB
|
||||
One can override these defaults by setting environment variables ORT_TENSORRT_MAX_BATCH_SIZE and ORT_TENSORRT_MAX_WORKSPACE_SIZE.
|
||||
e.g. on Linux
|
||||
|
||||
#### override default batch size to 10
|
||||
export ORT_TENSORRT_MAX_BATCH_SIZE=10
|
||||
|
||||
#### override default max workspace size to 2GB
|
||||
export ORT_TENSORRT_MAX_WORKSPACE_SIZE=2147483648
|
||||
|
||||
|
|
|
|||
|
|
@ -2,14 +2,15 @@
|
|||
<img src="docs/images/ngraph-logo.png">
|
||||
</p>
|
||||
|
||||
## nGraph Execution Provider (preview)
|
||||
## nGraph Execution Provider
|
||||
|
||||
[nGraph](https://github.com/NervanaSystems/ngraph) is a deep learning compiler from Intel®. The integration of nGraph as an execution provider (EP) into ONNX Runtime accelerates performance of ONNX model workloads across wide range of hardware offerings. Microsoft and Intel worked closely to integrate the nGraph EP with ONNX Runtime to showcase the benefits of quantization (int8). The nGraph EP leverages Intel® DL Boost and delivers performance increase with minimal loss of accuracy relative to FP32 ONNX models. With the nGraph EP, the ONNX Runtime delivers better inference performance across range of Intel hardware including Intel® Xeon® Processors compared to a generic CPU execution provider.
|
||||
|
||||
### Build nGraph execution provider
|
||||
Developers can now tap into the power of nGraph through ONNX Runtime to accelerate inference performance of ONNX models. Instructions for building the nGraph execution provider from the source is available [here](https://github.com/Microsoft/onnxruntime/blob/master/BUILD.md#nGraph).
|
||||
Developers can now tap into the power of nGraph through ONNX Runtime to accelerate inference performance of ONNX models. Instructions for building the nGraph execution provider from the source is available [here](../../BUILD.md#nGraph).
|
||||
|
||||
While the nGraph Compiler stack supports various operating systems and backends ([full list available here](https://www.ngraph.ai/ecosystem)), the nGraph execution provider for ONNX Runtime is validated for the following:
|
||||
|
||||
### Supported OS
|
||||
* Ubuntu 16.04
|
||||
* Windows 10 (`DEX_ONLY` mode is only one supported for the moment, codegen mode is work-in-progress.)
|
||||
|
|
@ -27,10 +28,12 @@ InferenceSession session_object{so};
|
|||
session_object.RegisterExecutionProvider(std::make_unique<::onnxruntime::NGRAPHExecutionProvider>());
|
||||
status = session_object.Load(model_file_name);
|
||||
```
|
||||
The C API details are [here](https://github.com/Microsoft/onnxruntime/blob/master/docs/C_API.md#c-api).
|
||||
The C API details are [here](../C_API.md#c-api).
|
||||
|
||||
### Python
|
||||
When using the python wheel from the ONNX Runtime built with nGraph execution provider, it will be automatically prioritized over the CPU execution provider. Python APIs details are [here](https://github.com/Microsoft/onnxruntime/blob/master/docs/python/api_summary.rst#api-summary).
|
||||
When using the python wheel from the ONNX Runtime built with nGraph execution provider, it will be automatically prioritized over the CPU execution provider. Python APIs details are [here](../python/api_summary.rst#api-summary).
|
||||
|
||||
### Using onnxruntime_perf_test
|
||||
You can test the performance of your ONNX Model with the nGraph execution provider. Use the flag -e ngraph in [onnxruntime_perf_test](https://github.com/Microsoft/onnxruntime/tree/master/onnxruntime/test/perftest#onnxruntime-performance-test).
|
||||
### Performance Tuning
|
||||
For performance tuning, please see guidance on this page: [ONNX Runtime Perf Tuning](../ONNX_Runtime_Perf_Tuning.md)
|
||||
|
||||
When/if using [onnxruntime_perf_test](../../onnxruntime/test/perftest), use the flag -e ngraph
|
||||
|
|
|
|||
Loading…
Reference in a new issue