saymrwulf/onnxruntime: ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-01 03:45:06 +00:00

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

Find a file

Tianlei Wu 5735e1bce0 Dump nodes with potential overflow in half conversion (#23363 ) Add a tool to generate node_block_list used in [float16 conversion tool](`04030f64be/onnxruntime/python/tools/transformers/float16.py (L175)`). Previously, we have a feature to dump statistics data (like min, max) of each node input/output. However, it is time consuming to generate a list of nodes that need to be kept in float32 when model is large. This could help speed up the process by outputting a list of nodes that have potential overflow in float-to-half conversion. Usage is to build onnxruntime from source with ` --cmake_extra_defines onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS=1`, then set some environment variables before running float32 optimized onnx model like: ``` export ORT_DEBUG_NODE_IO_DUMP_HALF_CONVERSION_OVERFLOW=1 export ORT_DEBUG_NODE_IO_HALF_OVERFLOW_THRESHOLD=50000 python benchmark.py -e optimum --height 1024 --width 1024 --steps 3 -b 1 -v Flux.1D -p flux1_dev_onnx/fp32_opt --skip_warmup ``` The threshold `ORT_DEBUG_NODE_IO_HALF_OVERFLOW_THRESHOLD` shall be <= 65504. The default value is 50000 if the environment variable is not set. It is better to leave some margin if number of samples are not large enough in the test. As a demo, we add an option --skip_warmup to benchmark.py for Flux, so that we can reduce the time on dumping warm-up runs. Example snippet of stdout (each inference session has such a summary when session ended): ``` Total counter in node dumping: 141 Found 2 nodes cannot be converted to half precision due to potential input/output overflow. Operator frequencies for these nodes: Softmax : 1 MatMul : 1 # ------- # Example python script for float16 conversion # For details, search `node_block_list` in https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/float16.py # ------- from onnxruntime.transformers.onnx_model import OnnxModel m = OnnxModel(onnx.load('flux1_dev_onnx/fp32_opt/vae_decoder/model.onnx')) node_block_list = [ '/decoder/mid_block/attentions.0/Softmax', '/decoder/mid_block/attentions.0/MatMul', ] m.convert_float_to_float16(keep_io_types=False, node_block_list=node_block_list) m.save_model_to_file('fp16/optimized.onnx', use_external_data_format=False) ``` Then you can use the python script to convert corresponding model to float16. ### Motivation and Context It is a tool used to generate node_block_list used in float16 conversion of stable diffusion 3.x and flux models in https://github.com/microsoft/onnxruntime/pull/22986. In stable diffusion or Flux pipeline, there are multiple models and there could be multiple session runs for each model. Without a proper tool, it is time consuming to get node_block_list for each model.		2025-01-16 12:54:46 -08:00
.config	Auto-generated baselines by 1ES Pipeline Templates (#22817 )	2024-11-13 13:50:52 -08:00
.devcontainer	Remove two lines in the Dockerfile for Github Codespace (#12278 )	2022-07-21 20:52:17 -07:00
.gdn	Update win-ci-pipeline.yml: enable xnnpack tests (#16244 )	2023-06-14 19:12:42 -07:00
.github	Update MACOSX_DEPLOYMENT_TARGET (#23308 )	2025-01-10 14:25:32 -08:00
.pipelines	[DML EP] Update DML to 1.15.4 (#22635 )	2024-10-29 17:13:57 -07:00
.vscode	Stop VSCode appending file associations to settings.json (#21944 )	2024-08-31 19:04:12 -07:00
cgmanifests	Update xnnpack, cpuinfo and pthreadpool (#23362 )	2025-01-15 09:42:15 -08:00
cmake	[WebGPU] allow build WebGPU EP for WebAssembly (#23364 )	2025-01-16 10:52:17 -08:00
csharp	[CoreML] Create EP by AppendExecutionProvider (#22675 )	2024-11-27 09:26:31 +08:00
dockerfiles	Update range of gpu arch (#23309 )	2025-01-14 14:27:34 -08:00
docs	Register opset 22 (#23344 )	2025-01-16 11:26:34 -08:00
include/onnxruntime/core	Add QNN EP HTP shared memory allocator (#23136 )	2025-01-14 11:09:50 -08:00
java	Revert DML pipeline changes (#23135 )	2024-12-18 10:42:10 -08:00
js	Update android_min_sdk_version/android_target_sdk_version (#23369 )	2025-01-16 08:03:31 -08:00
objectivec	Use UTF8 string encoding in ORTSaveCodeAndDescriptionToError(). (#22982 )	2024-12-02 17:41:52 -08:00
onnxruntime	Dump nodes with potential overflow in half conversion (#23363 )	2025-01-16 12:54:46 -08:00
orttraining	Use ruff as the formatter to replace black-isort (#23397 )	2025-01-16 11:14:15 -08:00
rust	Fix typos according to reviewdog report. (#21335 )	2024-07-22 13:37:32 -07:00
samples	Removed all the deprecated python training code and related tests and utils (#18333 )	2023-11-17 18:19:21 -08:00
tools	Use ruff as the formatter to replace black-isort (#23397 )	2025-01-16 11:14:15 -08:00
winml	Bump clang-format from 18.1.8 to 19.1.6 (#23346 )	2025-01-14 09:02:04 -08:00
.clang-format	Prevent GSL_SUPPRESS arguments from being modified by clang-format (#17242 )	2023-08-22 18:26:53 -07:00
.clang-tidy	Create clang-tidy CI (#12653 )	2022-09-30 08:05:38 -07:00
.dockerignore
.gitattributes	Fix typos according to reviewdog report. (#21335 )	2024-07-22 13:37:32 -07:00
.gitignore	Build onnxruntime.dll as arm64x (#18633 )	2023-12-06 16:49:00 -08:00
.gitmodules	Revert "Upgrade emsdk from 3.1.59 to 3.1.62" (#21817 )	2024-08-22 11:21:00 -07:00
.lintrunner.toml	Use ruff as the formatter to replace black-isort (#23397 )	2025-01-16 11:14:15 -08:00
build.bat	try to find patch.exe in git default installation folder (#17106 )	2023-08-10 21:48:13 -07:00
build.sh	Upgrade old Python version in packaging pipeline (#16667 )	2023-07-17 08:24:47 -07:00
build_arm64x.bat	remove unnecessary environment variable (#19166 )	2024-01-16 16:24:37 -08:00
CITATION.cff	Fix citation author name issue (#19597 )	2024-02-22 17:03:56 -08:00
CODEOWNERS	Update CODEOWNERS: remove onnxruntime-es (#21677 )	2024-12-17 13:39:13 -08:00
CONTRIBUTING.md	Fix link to High Level Design (#11786 )	2023-02-28 11:05:54 -08:00
CPPLINT.cfg	Ignore all whitespace lint messages for cpplint (#22781 )	2024-11-08 14:31:28 -08:00
lgtm.yml	Fix lgtm C++ error (#13613 )	2022-11-10 10:06:22 -08:00
LICENSE
NuGet.config	Update C# test projects (#21631 )	2024-09-05 08:21:23 +10:00
ort.wprp	Fully dynamic ETW controlled logging for ORT and QNN logs (#20537 )	2024-06-06 21:11:14 -07:00
ORT_icon_for_light_bg.png
packages.config	[DML EP] Update DML to 1.15.4 (#22635 )	2024-10-29 17:13:57 -07:00
pyproject.toml	Use ruff as the formatter to replace black-isort (#23397 )	2025-01-16 11:14:15 -08:00
README.md	Update pipeline status (#22924 )	2024-11-24 21:26:27 -08:00
requirements-dev.txt	Update python version metadata (remove 3.7, 3.8, 3.9; add 3.13). (#23067 )	2024-12-17 10:59:20 -08:00
requirements-doc.txt
requirements-lintrunner.txt	Use ruff as the formatter to replace black-isort (#23397 )	2025-01-16 11:14:15 -08:00
requirements-training.txt	ONNX 1.15 integration (#17125 )	2023-09-26 14:44:48 -07:00
requirements.txt	Add compatibility for NumPy 2.0 (#21085 )	2024-06-27 13:50:53 -07:00
SECURITY.md
setup.py	[MigraphX EP] [ROCm EP] Upstream ROCm changes for bugfixes and features (#23249 )	2025-01-15 12:57:04 -08:00
ThirdPartyNotices.txt	Cleanup code (#22827 )	2024-11-19 14:13:33 -08:00
VERSION_NUMBER	bumps up version in main from 1.20 -> 1.21 (#22482 )	2024-10-17 12:32:35 -07:00

README.md

ONNX Runtime is a cross-platform inference and training machine-learning accelerator.

ONNX Runtime inference can enable faster customer experiences and lower costs, supporting models from deep learning frameworks such as PyTorch and TensorFlow/Keras as well as classical machine learning libraries such as scikit-learn, LightGBM, XGBoost, etc. ONNX Runtime is compatible with different hardware, drivers, and operating systems, and provides optimal performance by leveraging hardware accelerators where applicable alongside graph optimizations and transforms. Learn more →

ONNX Runtime training can accelerate the model training time on multi-node NVIDIA GPUs for transformer models with a one-line addition for existing PyTorch training scripts. Learn more →

Get Started & Resources

General Information: onnxruntime.ai
Usage documentation and tutorials: onnxruntime.ai/docs
YouTube video tutorials: youtube.com/@ONNXRuntime
Upcoming Release Roadmap
Companion sample repositories:
- ONNX Runtime Inferencing: microsoft/onnxruntime-inference-examples
- ONNX Runtime Training: microsoft/onnxruntime-training-examples

Builtin Pipeline Status

System	Inference	Training
Windows
Linux
Mac
Android
iOS
Web
Other

This project is tested with BrowserStack.

Third-party Pipeline Status

System	Inference	Training
Linux

Releases

The current release and past releases can be found here: https://github.com/microsoft/onnxruntime/releases.

For details on the upcoming release, including release dates, announcements, features, and guidance on submitting feature requests, please visit the release roadmap: https://onnxruntime.ai/roadmap.

Data/Telemetry

Windows distributions of this project may collect usage data and send it to Microsoft to help improve our products and services. See the privacy statement for more details.

Contributions and Feedback

We welcome contributions! Please see the contribution guidelines.

For feature requests or bug reports, please file a GitHub Issue.

For general discussion or questions, please use GitHub Discussions.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

License

This project is licensed under the MIT License.