From fe3e672400d60c20baabcee35c9b9cb23733ee4b Mon Sep 17 00:00:00 2001 From: George Wu Date: Fri, 3 Dec 2021 21:48:12 -0800 Subject: [PATCH] updates to CUDA and TensorRT EP documentation (#9932) * updates to TRT, CUDA documentation * update trt options python example. --- docs/build/eps.md | 4 +-- .../CUDA-ExecutionProvider.md | 3 ++ .../TensorRT-ExecutionProvider.md | 34 ++++++++++++++++--- 3 files changed, 34 insertions(+), 7 deletions(-) diff --git a/docs/build/eps.md b/docs/build/eps.md index 9a9c16635a..0dcd9e5395 100644 --- a/docs/build/eps.md +++ b/docs/build/eps.md @@ -100,7 +100,7 @@ See more information on the TensorRT Execution Provider [here](../execution-prov * The path to the cuDNN installation (path to folder that contains libcudnn.so) must be provided via the cuDNN_PATH environment variable, or `--cudnn_home` parameter. * Install [TensorRT](https://developer.nvidia.com/tensorrt) * The TensorRT execution provider for ONNX Runtime is built and tested with TensorRT 8.0.3.4. - * To use earlier versions of TensorRT, prior to building, change the onnx-tensorrt submodule to a branch corresponding to the TensorRT version. e.g. To use TensorRT 7.2.x, + * To use different versions of TensorRT, prior to building, change the onnx-tensorrt submodule to a branch corresponding to the TensorRT version. e.g. To use TensorRT 7.2.x, * cd cmake/external/onnx-tensorrt * git remote update * git checkout 7.2.1 @@ -120,7 +120,7 @@ See more information on the TensorRT Execution Provider [here](../execution-prov ``` # to build with the latest supported TensorRT version ./build.sh --cudnn_home --cuda_home --use_tensorrt --tensorrt_home -# to build with earlier version. e.g. TensorRT 7.2.1 +# to build with different version. e.g. TensorRT 7.2.1 cd cmake/external/onnx-tensorrt git remote update git checkout 7.2.1 diff --git a/docs/execution-providers/CUDA-ExecutionProvider.md b/docs/execution-providers/CUDA-ExecutionProvider.md index d4607116f5..fe383ab598 100644 --- a/docs/execution-providers/CUDA-ExecutionProvider.md +++ b/docs/execution-providers/CUDA-ExecutionProvider.md @@ -25,6 +25,9 @@ Pre-built binaries of ONNX Runtime with CUDA EP are published for most language ## Requirements Please reference table below for official GPU packages dependencies for the ONNX Runtime inferencing package. Note that ONNX Runtime Training is aligned with PyTorch CUDA versions; refer to the Training tab on https://onnxruntime.ai/ for supported versions. +Note: Because of CUDA Minor Version Compatibility, Onnx Runtime built with CUDA 11.4 should be compatible with any CUDA 11.x version. +Please reference [Nvidia CUDA Minor Version Compatibility](https://docs.nvidia.com/deploy/cuda-compatibility/#minor-version-compatibility). + |ONNX Runtime|CUDA|cuDNN|Notes| |---|---|---|---| |1.10|11.4|8.2.4 (Linux)
8.2.2.26 (Windows)|libcudart 11.4.43
libcufft 10.5.2.100
libcurand 10.2.5.120
libcublasLt 11.6.1.51
libcublas 11.6.1.51
libcudnn 8.2.4| diff --git a/docs/execution-providers/TensorRT-ExecutionProvider.md b/docs/execution-providers/TensorRT-ExecutionProvider.md index ecc8a6c206..9777a81e84 100644 --- a/docs/execution-providers/TensorRT-ExecutionProvider.md +++ b/docs/execution-providers/TensorRT-ExecutionProvider.md @@ -26,6 +26,8 @@ Pre-built packages and Docker images are available for Jetpack in the [Jetson Zo |ONNX Runtime|TensorRT|CUDA| |---|---|---| +|master|8.2|11.4| +|1.10|8.0|11.4| |1.9|8.0|11.4| |1.7-1.8|7.2|11.0.3| |1.5-1.6|7.1|10.2| @@ -58,9 +60,13 @@ If some operators in the model are not supported by TensorRT, ONNX Runtime will ### Python -When using the Python wheel from the ONNX Runtime build with TensorRT execution provider, it will be automatically prioritized over the default GPU or CPU execution providers. There is no need to separately register the execution provider. -*Note that the next release (ORT 1.10) will require explicitly setting the providers parameter if you want to use execution providers other than the default CPU provider when instantiating InferenceSession.* - +To use TensorRT execution provider, you must explicitly register TensorRT execution provider when instantiating the InferenceSession. +Note that it is recommended you also register CUDAExecutionProvider to allow Onnx Runtime to assign nodes to CUDA execution provider that TensorRT does not support. +``` +import onnxruntime as ort +# set providers to ['TensorrtExecutionProvider', 'CUDAExecutionProvider'] with TensorrtExecutionProvider having the higher priority. +sess = ort.InferenceSession('model.onnx', providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider']) +``` ## Configurations There are two ways to configure TensorRT settings, either by environment variables or by execution provider option APIs. @@ -182,9 +188,27 @@ session_options.AppendExecutionProvider_TensorRT(trt_options); #### Python API example ``` +import onnxruntime as ort + +model_path = '' + +providers = [ + ('TensorrtExecutionProvider', { + 'device_id': 1, + 'trt_max_workspace_size': 2147483648, + 'trt_fp16_enable': True, + }), + ('CUDAExecutionProvider', { + 'device_id': 1, + 'arena_extend_strategy': 'kNextPowerOfTwo', + 'gpu_mem_limit': 2 * 1024 * 1024 * 1024, + 'cudnn_conv_algo_search': 'EXHAUSTIVE', + 'do_copy_in_default_stream': True, + }) +] + sess_opt = ort.SessionOptions() -sess = ort.InferenceSession('model.onnx', sess_options=sess_opt) -sess.set_providers(["TensorrtExecutionProvider"],[{'device_id': '1', 'trt_max_workspace_size': '2147483648', 'trt_fp16_enable':'True'}]) +sess = ort.InferenceSession(model_path, sess_options=sess_opt, providers=providers) ``` ## Performance Tuning