saymrwulf/onnxruntime: ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-02 03:55:34 +00:00

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

Find a file

Tianlei Wu 64819f6f8c Update benchmark_mha.py to compare with PyTorch SDPA (#21449 ) ### Description * Update benchmark_mha.py to compare with PyTorch SDPA api. * Write results to csv file. * Use sdpa_kernel cuda provider option instead of environment variables for better control. * Add arguments (`--use_gpu`, `--causal` etc) to allow testing different senarios. * Update benchmark_mha.sh to add cpu benchmarks For Q,K,V format, torch uses BNSH format, while ort uses BSNH format, so the result is not apple-to-apple. However, if the latency difference is large, that could be a warning. #### Example GPU results Example results on A100-SXM4-80GB with settings (use_gpu=TRUE, enable_cuda_graph=FALSE, causal=FALSE, past_sequence_length=0, intra_op_num_threads=0) in Azure Linux. ORT: build from source with CUDA 12.5; PyTorch 2.3.1 for cuda 12.1. format \| batch_size \| sequence_length \| num_heads \| head_size \| latency (s) \| tflops \| kernel -- \| -- \| -- \| -- \| -- \| -- \| -- \| -- Q,KV \| 4 \| 2048 \| 32 \| 128 \| 0.0015 \| 179.5 \| ort:flash Q,KV \| 4 \| 2048 \| 32 \| 128 \| 0.0015 \| 179.0 \| ort:default Q,K,V \| 4 \| 2048 \| 32 \| 128 \| 0.0016 \| 170.0 \| ort:default Q,K,V \| 4 \| 2048 \| 32 \| 128 \| 0.0016 \| 169.5 \| ort:flash QKV \| 4 \| 2048 \| 32 \| 128 \| 0.0016 \| 168.5 \| ort:default QKV \| 4 \| 2048 \| 32 \| 128 \| 0.0016 \| 167.4 \| ort:flash Q,K,V \| 4 \| 2048 \| 32 \| 128 \| 0.0017 \| 159.4 \| torch:default Q,K,V \| 4 \| 2048 \| 32 \| 128 \| 0.0018 \| 155.0 \| torch:flash Q,KV \| 4 \| 2048 \| 32 \| 128 \| 0.0030 \| 92.7 \| ort:efficient Q,K,V \| 4 \| 2048 \| 32 \| 128 \| 0.0030 \| 90.9 \| ort:efficient QKV \| 4 \| 2048 \| 32 \| 128 \| 0.0031 \| 89.9 \| ort:efficient Q,K,V \| 4 \| 2048 \| 32 \| 128 \| 0.0031 \| 89.0 \| torch:efficient Q,K,V \| 4 \| 2048 \| 32 \| 128 \| 0.0054 \| 51.3 \| torch:math Q,KV \| 4 \| 4096 \| 32 \| 128 \| 0.0058 \| 191.0 \| ort:default Q,KV \| 4 \| 4096 \| 32 \| 128 \| 0.0058 \| 190.6 \| ort:flash Q,K,V \| 4 \| 4096 \| 32 \| 128 \| 0.0059 \| 187.8 \| ort:default Q,K,V \| 4 \| 4096 \| 32 \| 128 \| 0.0059 \| 186.7 \| ort:flash QKV \| 4 \| 4096 \| 32 \| 128 \| 0.0059 \| 185.9 \| ort:flash QKV \| 4 \| 4096 \| 32 \| 128 \| 0.0059 \| 185.8 \| ort:default Q,K,V \| 4 \| 4096 \| 32 \| 128 \| 0.0067 \| 163.4 \| torch:default Q,K,V \| 4 \| 4096 \| 32 \| 128 \| 0.0070 \| 157.2 \| torch:flash Q,KV \| 4 \| 4096 \| 32 \| 128 \| 0.0113 \| 97.6 \| ort:efficient Q,K,V \| 4 \| 4096 \| 32 \| 128 \| 0.0114 \| 96.4 \| ort:efficient QKV \| 4 \| 4096 \| 32 \| 128 \| 0.0114 \| 96.2 \| ort:efficient Q,K,V \| 4 \| 4096 \| 32 \| 128 \| 0.0127 \| 86.3 \| torch:efficient Q,KV \| 8 \| 2048 \| 32 \| 128 \| 0.0031 \| 177.8 \| ort:flash Q,KV \| 8 \| 2048 \| 32 \| 128 \| 0.0031 \| 177.7 \| ort:default Q,K,V \| 8 \| 2048 \| 32 \| 128 \| 0.0032 \| 170.8 \| ort:default Q,K,V \| 8 \| 2048 \| 32 \| 128 \| 0.0032 \| 170.3 \| ort:flash QKV \| 8 \| 2048 \| 32 \| 128 \| 0.0032 \| 169.2 \| ort:default QKV \| 8 \| 2048 \| 32 \| 128 \| 0.0033 \| 169.0 \| ort:flash Q,K,V \| 8 \| 2048 \| 32 \| 128 \| 0.0034 \| 161.9 \| torch:default Q,K,V \| 8 \| 2048 \| 32 \| 128 \| 0.0036 \| 152.9 \| torch:flash Q,KV \| 8 \| 2048 \| 32 \| 128 \| 0.0059 \| 93.5 \| ort:efficient Q,K,V \| 8 \| 2048 \| 32 \| 128 \| 0.0060 \| 91.3 \| ort:efficient QKV \| 8 \| 2048 \| 32 \| 128 \| 0.0060 \| 91.0 \| ort:efficient Q,K,V \| 8 \| 2048 \| 32 \| 128 \| 0.0064 \| 86.0 \| torch:efficient Q,KV \| 8 \| 4096 \| 32 \| 128 \| 0.0115 \| 190.8 \| ort:flash Q,KV \| 8 \| 4096 \| 32 \| 128 \| 0.0115 \| 190.7 \| ort:default Q,K,V \| 8 \| 4096 \| 32 \| 128 \| 0.0118 \| 187.1 \| ort:default Q,K,V \| 8 \| 4096 \| 32 \| 128 \| 0.0118 \| 187.0 \| ort:flash QKV \| 8 \| 4096 \| 32 \| 128 \| 0.0118 \| 185.6 \| ort:default QKV \| 8 \| 4096 \| 32 \| 128 \| 0.0118 \| 185.6 \| ort:flash Q,K,V \| 8 \| 4096 \| 32 \| 128 \| 0.0139 \| 158.7 \| torch:default Q,K,V \| 8 \| 4096 \| 32 \| 128 \| 0.0139 \| 158.3 \| torch:flash Q,KV \| 8 \| 4096 \| 32 \| 128 \| 0.0225 \| 97.7 \| ort:efficient Q,K,V \| 8 \| 4096 \| 32 \| 128 \| 0.0227 \| 96.8 \| ort:efficient QKV \| 8 \| 4096 \| 32 \| 128 \| 0.0228 \| 96.3 \| ort:efficient Q,K,V \| 8 \| 4096 \| 32 \| 128 \| 0.0260 \| 84.5 \| torch:efficient #### Example CPU results Dell XPS 8960 with i9-13900 CPU (use_gpu=FALSE, causal=FALSE, past_sequence_length=0) in Windows. ORT: build from source with CUDA 12.5; PyTorch 2.3.1 for cuda 12.1. format \| causal \| batch_size \| seq_len \| num_heads \| head_size \| threads \| latency (s) \| kernel -- \| -- \| -- \| -- \| -- \| -- \| -- \| -- \| -- Q,K,V \| FALSE \| 1 \| 128 \| 32 \| 128 \| 8 \| 0.0005 \| ort:flash Q,K,V \| FALSE \| 1 \| 128 \| 32 \| 128 \| 0 \| 0.0009 \| ort:flash Q,K,V \| FALSE \| 1 \| 128 \| 32 \| 128 \| 0 \| 0.0009 \| ort:math Q,K,V \| FALSE \| 1 \| 128 \| 32 \| 128 \| 4 \| 0.0009 \| ort:flash Q,K,V \| FALSE \| 1 \| 128 \| 32 \| 128 \| 2 \| 0.0014 \| ort:flash Q,K,V \| FALSE \| 1 \| 128 \| 32 \| 128 \| 1 \| 0.0025 \| ort:flash Q,K,V \| FALSE \| 1 \| 128 \| 32 \| 128 \| 2 \| 0.0045 \| torch:default Q,K,V \| FALSE \| 1 \| 128 \| 32 \| 128 \| 24 \| 0.0046 \| torch:default Q,K,V \| FALSE \| 1 \| 128 \| 32 \| 128 \| 8 \| 0.0046 \| torch:default Q,K,V \| FALSE \| 1 \| 128 \| 32 \| 128 \| 4 \| 0.0046 \| torch:default Q,K,V \| FALSE \| 1 \| 128 \| 32 \| 128 \| 1 \| 0.0047 \| torch:default Q,K,V \| FALSE \| 1 \| 256 \| 32 \| 128 \| 0 \| 0.0019 \| ort:flash Q,K,V \| FALSE \| 1 \| 256 \| 32 \| 128 \| 8 \| 0.0019 \| ort:flash Q,K,V \| FALSE \| 1 \| 256 \| 32 \| 128 \| 0 \| 0.0022 \| ort:math Q,K,V \| FALSE \| 1 \| 256 \| 32 \| 128 \| 4 \| 0.0030 \| ort:flash Q,K,V \| FALSE \| 1 \| 256 \| 32 \| 128 \| 2 \| 0.0047 \| ort:flash Q,K,V \| FALSE \| 1 \| 256 \| 32 \| 128 \| 1 \| 0.0086 \| ort:flash Q,K,V \| FALSE \| 1 \| 256 \| 32 \| 128 \| 2 \| 0.0161 \| torch:default Q,K,V \| FALSE \| 1 \| 256 \| 32 \| 128 \| 4 \| 0.0162 \| torch:default Q,K,V \| FALSE \| 1 \| 256 \| 32 \| 128 \| 8 \| 0.0162 \| torch:default Q,K,V \| FALSE \| 1 \| 256 \| 32 \| 128 \| 24 \| 0.0165 \| torch:default Q,K,V \| FALSE \| 1 \| 256 \| 32 \| 128 \| 1 \| 0.0166 \| torch:default Q,K,V \| FALSE \| 1 \| 512 \| 32 \| 128 \| 8 \| 0.0077 \| ort:flash Q,K,V \| FALSE \| 1 \| 512 \| 32 \| 128 \| 0 \| 0.0091 \| ort:flash Q,K,V \| FALSE \| 1 \| 512 \| 32 \| 128 \| 0 \| 0.0099 \| ort:math Q,K,V \| FALSE \| 1 \| 512 \| 32 \| 128 \| 4 \| 0.0103 \| ort:flash Q,K,V \| FALSE \| 1 \| 512 \| 32 \| 128 \| 2 \| 0.0177 \| ort:flash Q,K,V \| FALSE \| 1 \| 512 \| 32 \| 128 \| 1 \| 0.0328 \| ort:flash Q,K,V \| FALSE \| 1 \| 512 \| 32 \| 128 \| 2 \| 0.0624 \| torch:default Q,K,V \| FALSE \| 1 \| 512 \| 32 \| 128 \| 4 \| 0.0624 \| torch:default Q,K,V \| FALSE \| 1 \| 512 \| 32 \| 128 \| 8 \| 0.0625 \| torch:default Q,K,V \| FALSE \| 1 \| 512 \| 32 \| 128 \| 24 \| 0.0626 \| torch:default Q,K,V \| FALSE \| 1 \| 512 \| 32 \| 128 \| 1 \| 0.0640 \| torch:default Q,K,V \| FALSE \| 1 \| 1024 \| 32 \| 128 \| 8 \| 0.0286 \| ort:flash Q,K,V \| FALSE \| 1 \| 1024 \| 32 \| 128 \| 0 \| 0.0317 \| ort:flash Q,K,V \| FALSE \| 1 \| 1024 \| 32 \| 128 \| 4 \| 0.0367 \| ort:flash Q,K,V \| FALSE \| 1 \| 1024 \| 32 \| 128 \| 0 \| 0.0391 \| ort:math Q,K,V \| FALSE \| 1 \| 1024 \| 32 \| 128 \| 2 \| 0.0656 \| ort:flash Q,K,V \| FALSE \| 1 \| 1024 \| 32 \| 128 \| 1 \| 0.1235 \| ort:flash Q,K,V \| FALSE \| 1 \| 1024 \| 32 \| 128 \| 24 \| 0.2482 \| torch:default Q,K,V \| FALSE \| 1 \| 1024 \| 32 \| 128 \| 2 \| 0.2483 \| torch:default Q,K,V \| FALSE \| 1 \| 1024 \| 32 \| 128 \| 4 \| 0.2483 \| torch:default Q,K,V \| FALSE \| 1 \| 1024 \| 32 \| 128 \| 8 \| 0.2486 \| torch:default Q,K,V \| FALSE \| 1 \| 1024 \| 32 \| 128 \| 1 \| 0.2538 \| torch:default Q,K,V \| FALSE \| 1 \| 2048 \| 32 \| 128 \| 0 \| 0.1038 \| ort:flash Q,K,V \| FALSE \| 1 \| 2048 \| 32 \| 128 \| 8 \| 0.1050 \| ort:flash Q,K,V \| FALSE \| 1 \| 2048 \| 32 \| 128 \| 0 \| 0.1368 \| ort:math Q,K,V \| FALSE \| 1 \| 2048 \| 32 \| 128 \| 4 \| 0.1535 \| ort:flash Q,K,V \| FALSE \| 1 \| 2048 \| 32 \| 128 \| 2 \| 0.2461 \| ort:flash Q,K,V \| FALSE \| 1 \| 2048 \| 32 \| 128 \| 1 \| 0.4724 \| ort:flash Q,K,V \| FALSE \| 1 \| 2048 \| 32 \| 128 \| 8 \| 0.9835 \| torch:default Q,K,V \| FALSE \| 1 \| 2048 \| 32 \| 128 \| 4 \| 0.9841 \| torch:default Q,K,V \| FALSE \| 1 \| 2048 \| 32 \| 128 \| 24 \| 0.9841 \| torch:default Q,K,V \| FALSE \| 1 \| 2048 \| 32 \| 128 \| 2 \| 0.9873 \| torch:default Q,K,V \| FALSE \| 1 \| 2048 \| 32 \| 128 \| 1 \| 0.9985 \| torch:default ### Motivation and Context To compare with PyTorch SDPA on CPU and CUDA latency.		2024-07-26 18:45:14 -07:00
.config
.devcontainer
.gdn	Update win-ci-pipeline.yml: enable xnnpack tests (#16244 )	2023-06-14 19:12:42 -07:00
.github	Allow cpplint to always be green (#21491 )	2024-07-25 15:57:30 -07:00
.pipelines	Update DirectML from 1.14.1 to 1.15.0 (#21323 )	2024-07-22 16:59:03 -07:00
.vscode	disable gemm f16 on CPU (#19744 )	2024-03-01 13:44:29 -08:00
cgmanifests	Update text formatting in generate_cgmanifest.py (#21489 )	2024-07-26 08:46:54 -07:00
cmake	OVEP - PR 1.19 (#21443 )	2024-07-24 23:45:31 -07:00
csharp	Update ruff and clang-format versions (#21479 )	2024-07-24 11:50:11 -07:00
dockerfiles	Fix typos according to reviewdog report. (#21335 )	2024-07-22 13:37:32 -07:00
docs	[DML EP] Register ReduceMin-20 (#20477 )	2024-07-25 17:06:30 -07:00
include/onnxruntime/core	Add QNN EP option context_node_name_prefix to set EPContext node name prefix (#21236 )	2024-07-26 16:56:44 -07:00
java	Fix typos according to reviewdog report. (#21335 )	2024-07-22 13:37:32 -07:00
js	[WebNN EP] Update argMax/argMin to adapt to latest spec (#21452 )	2024-07-25 17:07:01 -07:00
objectivec	Fix Objective-C static analysis warnings. (#20417 )	2024-04-24 11:48:29 -07:00
onnxruntime	Update benchmark_mha.py to compare with PyTorch SDPA (#21449 )	2024-07-26 18:45:14 -07:00
orttraining	Fix security issue #22016 #22017 #22018 (#21333 )	2024-07-25 08:25:22 +08:00
rust	Fix typos according to reviewdog report. (#21335 )	2024-07-22 13:37:32 -07:00
samples	Removed all the deprecated python training code and related tests and utils (#18333 )	2023-11-17 18:19:21 -08:00
tools	Separating all GPU stages into different Pipelines (#21521 )	2024-07-26 14:54:45 -07:00
winml	Update ruff and clang-format versions (#21479 )	2024-07-24 11:50:11 -07:00
.clang-format	Prevent GSL_SUPPRESS arguments from being modified by clang-format (#17242 )	2023-08-22 18:26:53 -07:00
.clang-tidy
.dockerignore
.gitattributes	Fix typos according to reviewdog report. (#21335 )	2024-07-22 13:37:32 -07:00
.gitignore	Build onnxruntime.dll as arm64x (#18633 )	2023-12-06 16:49:00 -08:00
.gitmodules	[js/web] optimize module export and deployment (#20165 )	2024-05-20 09:51:16 -07:00
.lintrunner.toml	CoreML: Aggregated changes to add all required ops for priority model (#21472 )	2024-07-26 08:29:33 +10:00
build.bat	try to find patch.exe in git default installation folder (#17106 )	2023-08-10 21:48:13 -07:00
build.sh	Upgrade old Python version in packaging pipeline (#16667 )	2023-07-17 08:24:47 -07:00
build_arm64x.bat	remove unnecessary environment variable (#19166 )	2024-01-16 16:24:37 -08:00
CITATION.cff	Fix citation author name issue (#19597 )	2024-02-22 17:03:56 -08:00
CODEOWNERS	Add owners for public facing API files (#15288 )	2023-03-30 17:16:15 -07:00
CONTRIBUTING.md
lgtm.yml
LICENSE
NuGet.config
ort.wprp	Fully dynamic ETW controlled logging for ORT and QNN logs (#20537 )	2024-06-06 21:11:14 -07:00
ORT_icon_for_light_bg.png
packages.config	Update DirectML from 1.14.1 to 1.15.0 (#21323 )	2024-07-22 16:59:03 -07:00
pyproject.toml	Ignore ruff rule `N813` (#21477 )	2024-07-24 17:48:22 -07:00
README.md	Update README.md (#18963 )	2024-01-03 17:26:25 -08:00
requirements-dev.txt	ONNX 1.15 integration (#17125 )	2023-09-26 14:44:48 -07:00
requirements-doc.txt
requirements-lintrunner.txt	Update ruff and clang-format versions (#21479 )	2024-07-24 11:50:11 -07:00
requirements-training.txt	ONNX 1.15 integration (#17125 )	2023-09-26 14:44:48 -07:00
requirements.txt	Add compatibility for NumPy 2.0 (#21085 )	2024-06-27 13:50:53 -07:00
SECURITY.md
setup.py	Migraphx ep windows build (#21284 )	2024-07-11 21:21:38 -07:00
ThirdPartyNotices.txt	Fix typos according to reviewdog report. (#21335 )	2024-07-22 13:37:32 -07:00
VERSION_NUMBER	Bump up version in main from 1.18.0 to 1.19.0 (#20489 )	2024-04-29 20:21:41 -07:00

README.md

ONNX Runtime is a cross-platform inference and training machine-learning accelerator.

ONNX Runtime inference can enable faster customer experiences and lower costs, supporting models from deep learning frameworks such as PyTorch and TensorFlow/Keras as well as classical machine learning libraries such as scikit-learn, LightGBM, XGBoost, etc. ONNX Runtime is compatible with different hardware, drivers, and operating systems, and provides optimal performance by leveraging hardware accelerators where applicable alongside graph optimizations and transforms. Learn more →

ONNX Runtime training can accelerate the model training time on multi-node NVIDIA GPUs for transformer models with a one-line addition for existing PyTorch training scripts. Learn more →

Get Started & Resources

General Information: onnxruntime.ai
Usage documentation and tutorials: onnxruntime.ai/docs
YouTube video tutorials: youtube.com/@ONNXRuntime
Upcoming Release Roadmap
Companion sample repositories:
- ONNX Runtime Inferencing: microsoft/onnxruntime-inference-examples
- ONNX Runtime Training: microsoft/onnxruntime-training-examples

Builtin Pipeline Status

System	Inference	Training
Windows
Linux
Mac
Android
iOS
Web
Other

Third-party Pipeline Status

System	Inference	Training
Linux

Data/Telemetry

Windows distributions of this project may collect usage data and send it to Microsoft to help improve our products and services. See the privacy statement for more details.

Contributions and Feedback

We welcome contributions! Please see the contribution guidelines.

For feature requests or bug reports, please file a GitHub Issue.

For general discussion or questions, please use GitHub Discussions.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

License

This project is licensed under the MIT License.