saymrwulf/onnxruntime: ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-30 20:18:08 +00:00

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

Find a file

Ivan Komarov 16b39e5b87 `symbolic_shape_infer.py`: Fix slicing a tensor that has a sympy.Min() in its shape (#14384 ) ### Description `_infer_Slice()` is a function (arguably the most complex one) in `symbolic_shape_infer.py` that infers the shape of the output of a `Slice` node. This commit fixes an edge case in `_infer_Slice()` caused by a SymPy quirk. When both the end of the slice (let's call it `e`) and the corresponding dimension of the sliced tensor (let's call it `dim`) are arbitrary symbolic expressions, `symbolic_shape_infer.py` [checks](`de7a868d5f/onnxruntime/python/tools/symbolic_shape_infer.py (L1728)`) if `e <= dim`. Comparing symbolic expressions is hard in general, so if the comparison fails, `symbolic_shape_infer.py` [gives up](`de7a868d5f/onnxruntime/python/tools/symbolic_shape_infer.py (L1734)`) and assumes that `e` is equal to `dim`. A failure of this sort currently happens for expressions of the form `Y - X >= 0` where `Y` contains a `sympy.Min()` (`symbolic_shape_infer.py` tries to rewrite `X <= Y` comparisons in various ways, and `Y - X >= 0` is [one of them](`de7a868d5f/onnxruntime/python/tools/symbolic_shape_infer.py (L1664)`)). An simple example to illustrate this: ```python >>> import sympy >>> X = sympy.Symbol('X', positive=True, integer=True) >>> >>> y1 = 9999 >>> Y1 = X + y1 - 5000 >>> bool(Y1 - X >= 0) True >>> >>> y2 = X + 4999 >>> Y2 = X + y2 - 5000 >>> bool(Y2 - X >= 0) True >>> >>> y3 = sympy.Min(y1, y2) >>> Y3 = X + y3 - 5000 >>> bool(Y3 - X >= 0) Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".../venv/lib/python3.9/site-packages/sympy/core/relational.py", line 511, in __bool__ raise TypeError("cannot determine truth value of Relational") TypeError: cannot determine truth value of Relational ``` If you assume that `X` is positive symbol (`symbolic_shape` [does assume](`de7a868d5f/onnxruntime/python/tools/symbolic_shape_infer.py (L2129)`) this for graph inputs), then both `Y1 >= X` and `Y2 >= X` holds, and SymPy can prove this. This means that `Y3 >= X` also holds (since `Y3` is essentially equal to either `Y1` or `Y2`, depending on the value of `X`), but this is too hard for SymPy to prove. I confirmed that this is still the case for the latest SymPy version (`1.11.1`). This commit tries to fix this edge case by slightly rewriting the expression containing `sympy.Min()`. I explain the details in the comments in `symbolic_shape_infer.py`, so I won't duplicate them in the PR description. ### Motivation and Context This sounds like a very contrived example, but it actually appeared in the wild when we tried to infer shapes for an ONNX graph exported from PyTorch that used relative-position multihead attention from Fairseq. The problematic line is [here](`7d050ada7d/fairseq/modules/espnet_multihead_attention.py (L192)`). In our codebase, we have something like `matrix_bd = matrix_bd[:, :, :, : matrix_ac.size(-1)]` before we add `matrix_ac` and `matrix_bd`. `matrix_bd` is itself a result of another slice, hence its shape contains `sympy.Min()`, and the SymPy weirdness described above prevents `symbolic_shape_infer.py` from correctly inferring the final shape of `matrix_bd`. Then `symbolic_shape_infer.py` explodes when we try to add `matrix_ac` and `matrix_bd`, because their shapes are not compatible. I added a small self-contained unit test to illustrate the problem. Without the fix, `slice_out_cropped` has shape `[N + Min(42, N + 21) - 22]`, and `input` has shape `[N]`, and we get this: ``` > python onnxruntime_test_python_symbolic_shape_infer.py ..................Cannot determine if 22 - N < 0 Unable to determine if N <= N + Min(42, N + 21) - 22, treat as equal E.... ====================================================================== ERROR: test_slice_of_min (__main__.TestSymbolicShapeInferenceForSlice) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/dfyz/onnxruntime/onnxruntime/test/python/onnxruntime_test_python_symbolic_shape_infer.py", line 460, in test_slice_of_min model = SymbolicShapeInference.infer_shapes(onnx.helper.make_model(graph_def)) File "/home/dfyz/onnxruntime/onnxruntime/test/python/../../python/tools/symbolic_shape_infer.py", line 2461, in infer_shapes raise Exception("Incomplete symbolic shape inference") Exception: Incomplete symbolic shape inference ---------------------------------------------------------------------- Ran 23 tests in 0.486s FAILED (errors=1) ``` With the fix, both tensors have shape `[N]`, and the test passes. --------- Co-authored-by: Ivan Komarov <dfyz@yandex-team.ru>		2023-02-23 15:32:37 +01:00
.config	Update tsaoptions.json: update the email alias (#13448 )	2022-10-26 15:56:16 -07:00
.devcontainer	Remove two lines in the Dockerfile for Github Codespace (#12278 )	2022-07-21 20:52:17 -07:00
.gdn	Update compliance tasks in python packaging pipeline and fix some compile warnings (#8471 )	2021-07-30 17:16:37 -07:00
.github	Upgrade doxygen to fix C API docs build issue (#13950 )	2023-02-03 09:43:29 -08:00
.pipelines	use python 3.9.7 in windowai packaging pipeline (#14766 )	2023-02-23 09:48:42 +08:00
.vscode	cpplint & Eager mode: refactor and add comments to empty_* functions, general lint cleanup in ort_aten (#12238 )	2022-07-20 11:47:57 -04:00
cgmanifests	Revert mimalloc from v2.0.9 to v2.0.3 (#14603 )	2023-02-07 09:58:25 -08:00
cmake	Migrating ORT Extensions from Git submodule to cmake FetchContent (#14298 )	2023-02-22 19:42:36 -08:00
csharp	restore opset18 test (#14677 )	2023-02-20 18:19:10 +08:00
dockerfiles	[Build] Fix arm64 Docker build (#14283 )	2023-01-30 16:25:19 -08:00
docs	support NeoX-style rotary embedding (#14785 )	2023-02-22 18:21:34 -08:00
include/onnxruntime/core	Enable Opset11 Sequence Ops on DirectML, and make the CPU implementations agnostic to backend EP (#14442 )	2023-02-21 18:08:28 -08:00
java	Update java/build.gradle to not use deprecated features that were removed in gradle 8.0. (#14733 )	2023-02-20 11:19:49 +08:00
js	Bump jszip from 3.7.1 to 3.8.0 in /js/web (#14536 )	2023-02-07 01:38:00 +00:00
objectivec	[objc] Fix parameter name in documentation. (#14330 )	2023-01-18 16:54:59 -08:00
onnxruntime	`symbolic_shape_infer.py`: Fix slicing a tensor that has a sympy.Min() in its shape (#14384 )	2023-02-23 15:32:37 +01:00
orttraining	Enable Opset11 Sequence Ops on DirectML, and make the CPU implementations agnostic to backend EP (#14442 )	2023-02-21 18:08:28 -08:00
package/rpm	Bump ORT version number (#14226 )	2023-01-26 12:33:47 -08:00
rust	Add rust bindings (#12606 )	2023-02-08 14:57:15 -08:00
samples	Format all python files under onnxruntime with black and isort (#11324 )	2022-04-26 09:35:16 -07:00
tools	Migrating ORT Extensions from Git submodule to cmake FetchContent (#14298 )	2023-02-22 19:42:36 -08:00
winml	remove device_id parameter out of ExecutionProvider::GetAllocator() (#14580 )	2023-02-13 10:01:07 -08:00
.clang-format
.clang-tidy	Create clang-tidy CI (#12653 )	2022-09-30 08:05:38 -07:00
.dockerignore
.flake8	Remove miscellaneous nuphar configs (#13070 )	2022-09-26 13:41:28 -07:00
.gitattributes
.gitignore	Add rust bindings (#12606 )	2023-02-08 14:57:15 -08:00
.gitmodules	Remove unused git submodules (#13830 )	2022-12-07 21:59:16 -08:00
build.amd64.1411.bat
build.bat
build.sh
CITATION.cff	Fix CITATION.cff and add automatic validation of your citation metadata (#10478 )	2022-04-13 10:03:52 -07:00
CODEOWNERS	Add cgmanifest file in codeowner list (#13042 )	2022-09-22 18:58:01 -07:00
CONTRIBUTING.md	Add instructions for previewing docs changes (#12528 )	2023-02-09 16:25:46 -08:00
lgtm.yml	Fix lgtm C++ error (#13613 )	2022-11-10 10:06:22 -08:00
LICENSE	Remove year from license (#6658 )	2021-02-12 00:25:56 -08:00
NuGet.config
ort.wprp
ORT_icon_for_light_bg.png	Update nuget icon (#10672 )	2022-03-01 09:11:03 -08:00
packages.config	[DML EP] Upgrade DML to 1.10.1 (#14433 )	2023-01-25 21:07:10 -08:00
pyproject.toml	Update pylint config to include valid short names (#13631 )	2022-11-14 10:00:25 -08:00
README.md	[Readme] Update table for build pipelines (#14618 )	2023-02-08 09:44:20 -08:00
requirements-dev.txt	Introduce parameterized as a dev dependency (#11364 )	2022-04-26 17:24:39 -07:00
requirements-doc.txt	Add auto doc gen for ORTModule API during CI build (#7046 )	2021-03-22 10:20:33 -07:00
requirements-training.txt	Remove protobuf pin from training requirements (#13695 )	2022-11-22 12:27:18 -08:00
requirements.txt.in	Add additional python requirements (#11522 )	2022-05-20 16:16:18 -07:00
SECURITY.md	Microsoft mandatory file (#11619 )	2022-05-25 13:56:10 -07:00
setup.py	Stable Diffusion CUDA optimizations Part 2 (#14597 )	2023-02-07 07:49:15 -08:00
ThirdPartyNotices.txt	Revert mimalloc from v2.0.9 to v2.0.3 (#14603 )	2023-02-07 09:58:25 -08:00
VERSION_NUMBER	Bump ORT version number (#14226 )	2023-01-26 12:33:47 -08:00

README.md

ONNX Runtime is a cross-platform inference and training machine-learning accelerator.

ONNX Runtime inference can enable faster customer experiences and lower costs, supporting models from deep learning frameworks such as PyTorch and TensorFlow/Keras as well as classical machine learning libraries such as scikit-learn, LightGBM, XGBoost, etc. ONNX Runtime is compatible with different hardware, drivers, and operating systems, and provides optimal performance by leveraging hardware accelerators where applicable alongside graph optimizations and transforms. Learn more →

ONNX Runtime training can accelerate the model training time on multi-node NVIDIA GPUs for transformer models with a one-line addition for existing PyTorch training scripts. Learn more →

Get Started & Resources

General Information: onnxruntime.ai
Usage documention and tutorials: onnxruntime.ai/docs
YouTube video tutorials: youtube.com/@ONNXRuntime
Upcoming Release Roadmap
Companion sample repositories:
- ONNX Runtime Inferencing: microsoft/onnxruntime-inference-examples
- ONNX Runtime Training: microsoft/onnxruntime-training-examples

Build Pipeline Status

System	Inference	Training
Windows
Linux
Mac
Android
iOS
Web
Other

Data/Telemetry

Windows distributions of this project may collect usage data and send it to Microsoft to help improve our products and services. See the privacy statement for more details.

Contributions and Feedback

We welcome contributions! Please see the contribution guidelines.

For feature requests or bug reports, please file a GitHub Issue.

For general discussion or questions, please use GitHub Discussions.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

License

This project is licensed under the MIT License.