saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-13 18:08:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

updates to CUDA and TensorRT EP documentation (#9932)

Browse source 
* updates to TRT, CUDA documentation

* update trt options python example.
This commit is contained in:
George Wu 2021-12-03 21:48:12 -08:00 committed by GitHub
parent 24dcd7334b
commit fe3e672400
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
3 changed files with 34 additions and 7 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
docs/build/eps.md vendored
View file

@ -100,7 +100,7 @@ See more information on the TensorRT Execution Provider [here](../execution-prov
* The path to the cuDNN installation (path to folder that contains libcudnn.so) must be provided via the cuDNN_PATH environment variable, or `--cudnn_home` parameter.
* Install [TensorRT](https://developer.nvidia.com/tensorrt)
* The TensorRT execution provider for ONNX Runtime is built and tested with TensorRT 8.0.3.4.
* To use earlier versions of TensorRT, prior to building, change the onnx-tensorrt submodule to a branch corresponding to the TensorRT version. e.g. To use TensorRT 7.2.x,
* To use different versions of TensorRT, prior to building, change the onnx-tensorrt submodule to a branch corresponding to the TensorRT version. e.g. To use TensorRT 7.2.x,
* cd cmake/external/onnx-tensorrt
* git remote update
* git checkout 7.2.1
@ -120,7 +120,7 @@ See more information on the TensorRT Execution Provider [here](../execution-prov
```
# to build with the latest supported TensorRT version
./build.sh --cudnn_home <path to cuDNN e.g. /usr/lib/x86_64-linux-gnu/> --cuda_home <path to folder for CUDA e.g. /usr/local/cuda> --use_tensorrt --tensorrt_home <path to TensorRT home>
# to build with earlier version. e.g. TensorRT 7.2.1
# to build with different version. e.g. TensorRT 7.2.1
cd cmake/external/onnx-tensorrt
git remote update
git checkout 7.2.1

3
docs/execution-providers/CUDA-ExecutionProvider.md
View file

@ -25,6 +25,9 @@ Pre-built binaries of ONNX Runtime with CUDA EP are published for most language
## Requirements
Please reference table below for official GPU packages dependencies for the ONNX Runtime inferencing package. Note that ONNX Runtime Training is aligned with PyTorch CUDA versions; refer to the Training tab on https://onnxruntime.ai/ for supported versions.
Note: Because of CUDA Minor Version Compatibility, Onnx Runtime built with CUDA 11.4 should be compatible with any CUDA 11.x version.
Please reference [Nvidia CUDA Minor Version Compatibility](https://docs.nvidia.com/deploy/cuda-compatibility/#minor-version-compatibility).
|ONNX Runtime|CUDA|cuDNN|Notes|
|---|---|---|---|
|1.10|11.4|8.2.4 (Linux)<br/>8.2.2.26 (Windows)|libcudart 11.4.43<br/>libcufft 10.5.2.100<br/>libcurand 10.2.5.120<br/>libcublasLt 11.6.1.51<br/>libcublas 11.6.1.51<br/>libcudnn 8.2.4|

34
docs/execution-providers/TensorRT-ExecutionProvider.md
View file

@ -26,6 +26,8 @@ Pre-built packages and Docker images are available for Jetpack in the [Jetson Zo
|ONNX Runtime|TensorRT|CUDA|
|---|---|---|
|master|8.2|11.4|
|1.10|8.0|11.4|
|1.9|8.0|11.4|
|1.7-1.8|7.2|11.0.3|
|1.5-1.6|7.1|10.2|
@ -58,9 +60,13 @@ If some operators in the model are not supported by TensorRT, ONNX Runtime will
### Python
When using the Python wheel from the ONNX Runtime build with TensorRT execution provider, it will be automatically prioritized over the default GPU or CPU execution providers. There is no need to separately register the execution provider.
*Note that the next release (ORT 1.10) will require explicitly setting the providers parameter if you want to use execution providers other than the default CPU provider when instantiating InferenceSession.*
To use TensorRT execution provider, you must explicitly register TensorRT execution provider when instantiating the InferenceSession.
Note that it is recommended you also register CUDAExecutionProvider to allow Onnx Runtime to assign nodes to CUDA execution provider that TensorRT does not support.
```
import onnxruntime as ort
# set providers to ['TensorrtExecutionProvider', 'CUDAExecutionProvider'] with TensorrtExecutionProvider having the higher priority.
sess = ort.InferenceSession('model.onnx', providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider'])
```
## Configurations
There are two ways to configure TensorRT settings, either by environment variables or by execution provider option APIs.
@ -182,9 +188,27 @@ session_options.AppendExecutionProvider_TensorRT(trt_options);
#### Python API example
```
import onnxruntime as ort
model_path = '<path to model>'
providers = [
('TensorrtExecutionProvider', {
'device_id': 1,
'trt_max_workspace_size': 2147483648,
'trt_fp16_enable': True,
}),
('CUDAExecutionProvider', {
'device_id': 1,
'arena_extend_strategy': 'kNextPowerOfTwo',
'gpu_mem_limit': 2 * 1024 * 1024 * 1024,
'cudnn_conv_algo_search': 'EXHAUSTIVE',
'do_copy_in_default_stream': True,
})
]
sess_opt = ort.SessionOptions()
sess = ort.InferenceSession('model.onnx', sess_options=sess_opt)
sess.set_providers(["TensorrtExecutionProvider"],[{'device_id': '1', 'trt_max_workspace_size': '2147483648', 'trt_fp16_enable':'True'}])
sess = ort.InferenceSession(model_path, sess_options=sess_opt, providers=providers)
```
## Performance Tuning