saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-13 18:08:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

[TensorRT EP] Update instruction of user_compute_stream (#20343)

Browse source 
### Description
<!-- Describe your changes. -->
Update instruction and add sample code

### Preview the change:

https://yf711.github.io/onnxruntime/docs/execution-providers/TensorRT-ExecutionProvider.html#execution-provider-options

### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
This commit is contained in:
Yifan Li 2024-04-17 13:45:06 -07:00 committed by GitHub
parent d3551b0069
commit 55edff104f
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 27 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

28
docs/execution-providers/TensorRT-ExecutionProvider.md
View file

@ -114,8 +114,34 @@ TensorRT configurations can be set by execution provider options. It's useful wh
* Default value: 0
* `user_compute_stream`: Defines the compute stream for the inference to run on. It implicitly sets the `has_user_compute_stream` option. It cannot be set through `UpdateTensorRTProviderOptions`, but rather `UpdateTensorRTProviderOptionsWithValue`.
* This cannot be used in combination with an external allocator.
* This can not be set using the python API.
* This can also be set using the python API.
* i.e The cuda stream captured from pytorch can be passed into ORT-TRT. Click below to check sample code:
<Details>
```python
import onnxruntime as ort
import torch
...
sess = ort.InferenceSession('model.onnx')
if torch.cuda.is_available():
s = torch.cuda.Stream()
option = {"user_compute_stream": str(s.cuda_stream)}
sess.set_providers(["TensorrtExecutionProvider"], [option])
options = sess.get_provider_options()
assert "TensorrtExecutionProvider" in options
assert options["TensorrtExecutionProvider"].get("user_compute_stream", "") == str(s.cuda_stream)
assert options["TensorrtExecutionProvider"].get("has_user_compute_stream", "") == "1"
...
```
</Details>
* To take advantage of user compute stream, it is recommended to use [I/O Binding](https://onnxruntime.ai/docs/api/python/api_summary.html#data-on-device) to bind inputs and outputs to tensors in device.
* `trt_max_workspace_size`: maximum workspace size for TensorRT engine.
* Default value: 1073741824 (1GB).