b53038b6a0
### Description 1. ALIGN_BYTES is set to 16 before because float4 is used for vectorization by default. This PR computes ALIGN_BYTES by vectorize size. 2. Fix wrong data access when using small elemant size (e.g., 1, 33). Small case may be used for SoftmaxTunableOp. 3. Fix the bug that data may be written first and then read in BlockReduce function on ROCm EP. There is a slightly performance improvement because all theads in warp-0 work. BlockReduce method before this PR: One block has N(warps_per_block) warps, one warp has M(WARP_SIZE) threads. step1. All the threads in one block read data into shared memory. step2. Reduce all data to the first warp. Only the first N threads of warp-0 are used. thread-0 computes data in warp-0 and writes the result into the location of data0, thread-1 computes data in warp-1 and writes the result into the location of data1. __syncwarp(mask) is necessary here to make sure thread-1,...N will delay writing data into warp-0 until thread-0 has finished reading data from warp-0. step3. Thread-0 reduces all vaild data(only the first N data) in warp-0 and writes the results into the location of data0, then return data0. Issue: ROCm doesn't support __syncwarp() now, we need another implementation to make sure read before write in warp-0. BlockReduce function in this PR. step2. Reduce all data to the first warp. Only the threads of warp-0 are used. Each thread in warp-0 read data from the same location of every warp and computes result. For example, thread-0 computes the first data of every warp and writes the result into the location of data0. step3. Thread-0 reduces all data in warp-0 and writes the results into the location of data0, then return data0. Shared memory  Test: kernel explorer will use small element to test. (https://github.com/microsoft/onnxruntime/pull/14541) |
||
|---|---|---|
| .config | ||
| .devcontainer | ||
| .gdn | ||
| .github | ||
| .pipelines | ||
| .vscode | ||
| cgmanifests | ||
| cmake | ||
| csharp | ||
| dockerfiles | ||
| docs | ||
| include/onnxruntime/core | ||
| java | ||
| js | ||
| objectivec | ||
| onnxruntime | ||
| orttraining | ||
| package/rpm | ||
| rust | ||
| samples | ||
| test | ||
| tools | ||
| winml | ||
| .clang-format | ||
| .clang-tidy | ||
| .dockerignore | ||
| .flake8 | ||
| .gitattributes | ||
| .gitignore | ||
| .gitmodules | ||
| build.amd64.1411.bat | ||
| build.bat | ||
| build.sh | ||
| CITATION.cff | ||
| CODEOWNERS | ||
| CONTRIBUTING.md | ||
| lgtm.yml | ||
| LICENSE | ||
| NuGet.config | ||
| ort.wprp | ||
| ORT_icon_for_light_bg.png | ||
| packages.config | ||
| pyproject.toml | ||
| README.md | ||
| requirements-dev.txt | ||
| requirements-doc.txt | ||
| requirements-training.txt | ||
| requirements.txt.in | ||
| SECURITY.md | ||
| setup.py | ||
| ThirdPartyNotices.txt | ||
| VERSION_NUMBER | ||
ONNX Runtime is a cross-platform inference and training machine-learning accelerator.
ONNX Runtime inference can enable faster customer experiences and lower costs, supporting models from deep learning frameworks such as PyTorch and TensorFlow/Keras as well as classical machine learning libraries such as scikit-learn, LightGBM, XGBoost, etc. ONNX Runtime is compatible with different hardware, drivers, and operating systems, and provides optimal performance by leveraging hardware accelerators where applicable alongside graph optimizations and transforms. Learn more →
ONNX Runtime training can accelerate the model training time on multi-node NVIDIA GPUs for transformer models with a one-line addition for existing PyTorch training scripts. Learn more →
Get Started & Resources
-
General Information: onnxruntime.ai
-
Usage documention and tutorials: onnxruntime.ai/docs
-
YouTube video tutorials: youtube.com/@ONNXRuntime
-
Companion sample repositories:
- ONNX Runtime Inferencing: microsoft/onnxruntime-inference-examples
- ONNX Runtime Training: microsoft/onnxruntime-training-examples
Build Pipeline Status
| System | Inference | Training |
|---|---|---|
| Windows | ||
| Linux | ||
| Mac | ||
| Android | ||
| iOS | ||
| Web | ||
| Other |
Data/Telemetry
Windows distributions of this project may collect usage data and send it to Microsoft to help improve our products and services. See the privacy statement for more details.
Contributions and Feedback
We welcome contributions! Please see the contribution guidelines.
For feature requests or bug reports, please file a GitHub Issue.
For general discussion or questions, please use GitHub Discussions.
Code of Conduct
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.
License
This project is licensed under the MIT License.