mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-06-08 00:23:03 +00:00
6115c8fd1f
This PR makes ORT support TRT plugin using custom ops. ORT TRT can automatically register all TRT plugins from TRT plugins registry as custom ops. There is no code change needed for ORT when new TRT plugins are introduced. Previous way for ORT to support TRT plugins was using contrib ops, but there are some concerns about it: - Contrib ops are shipped as part of the ORT binary by default. TRT related plugins should not be in the default ORT. - Contrib ops are designed for internal ops and developed for cpu and cuda EPs. Therefore, using custom ops is a good approach to support TRT plugins. Followings are the major modifications: 1. Add new `GetCustomOpDomainList` provider api which allows provider to create its own custom op domain list and ORT can register this domain list. Provider has the responsibility to free all the custom op domain instances it created. 2. Move OrtCustomOpDomain struct definition to framework_provider_common.h since this struct is being used by framework and EPs now. 3. There are several TRT plugins registered as onnx schema op through contrib op with onnx domain. In order not to break the old models using those TRT plugins which were registered with ONNX domain and maintain backward compatible, we need to keep the old/legacy TRT plugins with onnx domain. Moving forward, all newly added TRT plugins should be registered with `trt.plugins` domain. 4. TRT plugin doesn't have an api to get number of inputs/outputs of the registered plugins, so ORT TRT uses variadic inputs/outputs to bypass the onnx node validation. 5. Add new trt provider option, `trt_extra_plugin_lib_paths`, user can specify any extra plugin lib, for example, `fastertransformer/build/lib/libvit_plugin.so` or `fastertransformer/build/lib/libvit_plugin.so;fastertransformer/build/lib/libvit_plugin_v2.so` |
||
|---|---|---|
| .. | ||
| backend | ||
| datasets | ||
| providers/tvm | ||
| tools | ||
| torch_cpp_extensions | ||
| training | ||
| __init__.py | ||
| _ld_preload.py | ||
| _pybind_state.py.in | ||
| exported_symbols.lst | ||
| numpy_helper.h | ||
| onnxruntime_collect_build_info.py | ||
| onnxruntime_inference_collection.py | ||
| onnxruntime_pybind.h | ||
| onnxruntime_pybind_exceptions.cc | ||
| onnxruntime_pybind_exceptions.h | ||
| onnxruntime_pybind_iobinding.cc | ||
| onnxruntime_pybind_mlvalue.cc | ||
| onnxruntime_pybind_mlvalue.h | ||
| onnxruntime_pybind_module.cc | ||
| onnxruntime_pybind_ortvalue.cc | ||
| onnxruntime_pybind_schema.cc | ||
| onnxruntime_pybind_sparse_tensor.cc | ||
| onnxruntime_pybind_state.cc | ||
| onnxruntime_pybind_state.h | ||
| onnxruntime_pybind_state_common.cc | ||
| onnxruntime_pybind_state_common.h | ||
| onnxruntime_validation.py | ||
| pybind.def | ||
| version_script.lds | ||
| version_script_expose_onnx_protobuf.lds | ||