e8b0ae8923
* updated cmake files for trt * added trt execution provider * added trt basic test * removed trt_path action attribute * Add files via upload * Update build.py * Update trt_allocator.h * fixed issues found by reviewers * changed cast operator * added comment for custom kernel implementation * changed auto to auto& * changed to function compile APIs for TRT execution provider * changed to function compile APIs for TRT execution provider * added new DType DInt64 * adapted to the changes of onnxruntime_c_api * removed trt kernel (use function compile instead) * updated onnx-tensorrt submodule * set default memory type to TRT fused kernel * resolve merge conflict * fixed the issue that USE_CUDA conflicts with USE_TRT * construct graph by adding nodes in topological order * made changes for Windows * change buffers type * bypass HasImplementationOf check for TRT XP because TRT kernel is not registered * added domain to version info in rebuilt model proto * added trt to test option list * added DomainToVersionMap() to GraphViewer * removed Copy() * fixed broken code * format the code to clang format * used local reference to the frequently used values * fixed a couple of issues according to reviewers feedback * fixed a couple of issues according to reviewers feedback * added python binding for TRT and enable use_cuda when use_trt is on * fixed a redefinition issue * changed shared_ptr to unique_ptr on trt engines, and made a few changes required by reviewers * enabled trtexecution provider for unit tests * renamed trt to tensorrt * added tesorrt to python binding * update submodule onnx and onnx-tensorrt * made a couple of minor changes based on reviewer's feedback * added CUDA_CHECK * removed test code * fixed broken code after merge * updated onnx-tensorrt submodule * added post processing to align trt inputs/outputs with graph inputs/outputs * updated onnx submodule * added CUDA fallback for TensorRT and fixed TensorRT cmake issue * added ci pipeline for tensorrt and removed some redundent code from trt xp * fixed syntax issue * updated onnx-tensorrt submodule * fix trt build problem by: (#602) 1. Add additional /wd for debug build 2. Add io.h for additional targets 3. Bring back mb version of getopt * Update install_ubuntu.sh * Update linux-gpu-tensorrt-ci-pipeline.yml * Update linux-gpu-tensorrt-ci-pipeline.yml * Update run_build.sh * Update run_build.sh * Update run_build.sh * Update run_build.sh * fixed the issue that GetKernelRegistry returns nullptr * merged master to this branch * moved some data types to private * fixed tensorrt CI pipeline issue * customized test data for TensorRT pipeline * added onnx-tensorrt in json file and fixed an issue in ci script * added comments |
||
|---|---|---|
| .github/ISSUE_TEMPLATE | ||
| cmake | ||
| csharp | ||
| dockerfiles | ||
| docs | ||
| include/onnxruntime/core | ||
| onnxruntime | ||
| package/rpm | ||
| tools | ||
| .clang-format | ||
| .clang-tidy | ||
| .gitattributes | ||
| .gitignore | ||
| .gitmodules | ||
| build.amd64.1411.bat | ||
| build.bat | ||
| BUILD.md | ||
| build.sh | ||
| cgmanifest.json | ||
| CODEOWNERS | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| README.md | ||
| rename_manylinux.sh | ||
| requirements-dev.txt | ||
| requirements-doc.txt | ||
| requirements.txt | ||
| setup.py | ||
| ThirdPartyNotices.txt | ||
| VERSION_NUMBER | ||
ONNX Runtime is an open-source scoring engine for Open Neural Network Exchange (ONNX) models.
ONNX is an open format for machine learning (ML) models that is supported by various ML and DNN frameworks and tools. This format makes it easier to interoperate between frameworks and to maximize the reach of your hardware optimization investments. Learn more about ONNX on https://onnx.ai or view the Github Repo.
Why use ONNX Runtime
ONNX Runtime is an open architecture that is continually evolving to adapt to and address the newest developments and challenges in AI and Deep Learning. We will keep ONNX Runtime up to date with the ONNX standard, supporting all ONNX releases with future compatibliity while maintaining backwards compatibility with prior releases.
ONNX Runtime continuously strives to provide top performance for a broad and growing number of usage scenarios in Machine Learning. Our investments focus on these 3 core areas:
- Run any ONNX model
- High performance
- Cross platform
Run any ONNX model
Alignment with ONNX Releases
ONNX Runtime provides comprehensive support of the ONNX spec and can be used to run all models based on ONNX v1.2.1 and higher. See ONNX version release details here.
As of March 2019, ONNX Runtime supports ONNX 1.4.
Traditional ML support
ONNX Runtime fully supports the ONNX-ML profile of the ONNX spec for traditional ML scenarios.
High Performance
You can use ONNX Runtime with both CPU and GPU hardware. You can also plug in additional execution providers to ONNX Runtime. With many graph optimizations and various accelerators, ONNX Runtime can often provide lower latency and higher efficiency compared to other runtimes. This provides smoother end-to-end customer experiences and lower costs from improved machine utilization.
Currently ONNX Runtime supports CUDA, MLAS (Microsoft Linear Algebra Subprograms), MKL-DNN, and MKL-ML for computation acceleration. See more details on available build options here or refer to this page to add a new execution provider.
We are continuously working to integrate new execution providers to provide improvements in latency and efficiency. We have ongoing collaborations to integrate the following with ONNX Runtime:
- Intel MKL-DNN and nGraph
- NVIDIA TensorRT
Cross Platform
ONNX Runtime offers:
- APIs for Python, C#, and C
- Available for Linux, Windows, and Mac
See API documentation and package installation instructions below.
Looking ahead: To broaden the reach of the runtime, we will continue investments to make ONNX Runtime available and compatible with more platforms. If you have specific scenarios that are not currently supported, please share your suggestions via Github Issues.
Getting Started
If you need a model:
- Check out the ONNX Model Zoo for ready-to-use pre-trained models.
- To get an ONNX model by exporting from various frameworks, see ONNX Tutorials.
If you already have an ONNX model, just install the runtime for your machine to try it out. One easy way to deploy the model on the cloud is by using Azure Machine Learning. See detailed instructions and sample notebooks.
Installation
APIs and Official Builds
| API Documentation | CPU package | GPU package* |
|---|---|---|
| Python** | Available on Pypi
|
Available on Pypi
|
| C# | Available on Nuget
|
Available on Nuget
|
| C | Available on Nuget
Files (.zip, .tgz)
|
Available on Nuget
Files (.zip, .tgz)
|
| C++ | Build from source | Build from source |
*Requires CUDA 9.1 and cuDNN 7.3
**Compatible with Python 3.5-3.7
System Requirements
- ONNX Runtime binaries in CPU packages use OpenMP and depends on the library being available at runtime in the
system.
- For Windows, OpenMP support comes as part of VC runtime. It is also available as redist packages: vc_redist.x64.exe and vc_redist.x86.exe
- For Linux, the system must have the libgomp.so.1 which can be installed using
apt-get install libgomp1.
- The GPU builds require the CUDA9.1 and cuDNN 7.3 runtime libraries being installed in the system.
Build Details
For details on the build configurations and information on how to create a build, see Build ONNX Runtime.
Versioning
See more details on API and ABI Versioning and ONNX Compatibility in Versioning.
Design and Key Features
For an overview of the high level architecture and key decisions in the technical design of ONNX Runtime, see Engineering Design.
ONNX Runtime is built with an extensible design that makes it versatile to support a wide array of models with high performance.
- Add a custom operator/kernel
- Add an execution provider
- Add a new graph transform
- Add a new rewrite rule
Contribute
We welcome your contributions! Please see the contribution guidelines.
Feedback
For any feedback or to report a bug, please file a GitHub Issue.
Code of Conduct
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.