saymrwulf/onnxruntime: ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-28 20:11:22 +00:00

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

Find a file

pengwa 947aab0ae0 Make HF converge with lighting native amp (#13616 ) ### Fix training convergence issues #### Problem: Huggingface Transformers: 4.22.0 PyTorch Lightning: 1.6.3 PyTorch: v1.12.1, cuda 11.6 ORT: main branch, cuda 11.6 Model: RobertaForSequenceClassification @ models/roberta/modeling_roberta.py Mixed Precision training with `torch.autocast`: `a64e1dfd7d/pytorch_lightning/plugins/precision/native_amp.py (L99)` Under this amp autocast context, forward + loss computation run. Here is a snippet of loss computation. ``` if labels is not None: ... if self.config.problem_type == "regression": loss_fct = MSELoss() if self.num_labels == 1: ... elif self.config.problem_type == "single_label_classification": loss_fct = CrossEntropyLoss() loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) elif self.config.problem_type == "multi_label_classification": ... return SequenceClassifierOutput( loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions, ) ``` It is found after forward run, loss is 1.0850 in float16, looks good.. Then it did a scaling up here: `a64e1dfd7d/pytorch_lightning/plugins/precision/native_amp.py (L62)`, the scaler is 65536. then we get a scaled loss 71104 in float type (because float16 loss multiple fp32 scaler, type got promoted to fp32). Then backward started with initial grads to be 1, then 1 (float32) * 65536 (float32) as the backward step, generating a float16 gradient, then we got a `inf`. The problem occurs. With `inf`, the backward feed the `inf` into crossentropygradient op, generating `nan`s. Then all gradients got `nan` in back propagation. So we see training with ORTModule (it almost always `overflow`, the loss did not drop too much, as compared with PyTorch). #### Analysis for the UT (when autocast enabled) PyTorch trace graph looks like this : ``` graph(%0 : Float(16, 3, strides=[3, 1], requires_grad=0, device=cuda:0), %target : Long(16, strides=[1], requires_grad=0, device=cuda:0), %2 : Float(3, 3, strides=[3, 1], requires_grad=1, device=cuda:0)): %9 : int = prim::Constant[value=5]() # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/modules/linear.py:114:0 %10 : bool = prim::Constant[value=0]() # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/modules/linear.py:114:0 %11 : bool = prim::Constant[value=0]() # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/modules/linear.py:114:0 %12 : NoneType = prim::Constant() %13 : Half(3, 3, strides=[3, 1], requires_grad=0, device=cuda:0) = aten::to(%2, %9, %10, %11, %12) # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/modules/linear.py:114:0 %14 : int = prim::Constant[value=5]() # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/modules/linear.py:114:0 %15 : bool = prim::Constant[value=0]() # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/modules/linear.py:114:0 %16 : bool = prim::Constant[value=0]() # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/modules/linear.py:114:0 %17 : NoneType = prim::Constant() %18 : Half(16, 3, strides=[3, 1], requires_grad=0, device=cuda:0) = aten::to(%0, %14, %15, %16, %17) # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/modules/linear.py:114:0 %19 : NoneType = prim::Constant() %input : Half(16, 3, strides=[3, 1], requires_grad=0, device=cuda:0) = aten::linear(%18, %13, %19) # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/modules/linear.py:114:0 %21 : NoneType = prim::Constant() %22 : int = prim::Constant[value=1]() # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/functional.py:3,014:0 %23 : int = prim::Constant[value=-100]() # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/functional.py:3,014:0 %24 : float = prim::Constant[value=0.]() # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/functional.py:3,014:0 %data : Float(requires_grad=0, device=cuda:0) = aten::cross_entropy_loss(%input, %target, %21, %22, %23, %24) # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/functional.py:3,014:0 %27 : Float(requires_grad=0, device=cuda:0) = ^_OutputIdentityOp()(%data) # /opt/conda/envs/ptca/lib/python3.8/site-packages/onnxruntime/training/ortmodule/_io.py:430:0 return (%27) ``` The most important lines %target : Long(16, strides=[1], requires_grad=0, device=cuda:0), %input : _Half_(16, 3, strides=[3, 1], requires_grad=0, device=cuda:0) = aten::linear(%18, %13, %19) # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/modules/linear.py:114:0 _Float_(requires_grad=0, device=cuda:0) = aten::cross_entropy_loss(%_input_, %target, %21, %22, %23, %24) # /opt/conda/envs/ptca/lib/python3.8/site-packages/torch/nn/functional.py:3,014:0 `aten::cross_entropy_loss` takes Half input, and return Float output. As said in doc: https://pytorch.org/docs/stable/amp.html#cuda-ops-that-can-autocast-to-float32, `cross_entropy` in autocast mode will run in fp32 mode, e.g. convert its input to fp32 (if it is not), do the compute and return fp32 result. The other hand, ORT's `SoftmaxCrossEntropyLossInternal` take same types of input and output, and our code `31cb3cb254/orttraining/orttraining/python/training/ortmodule/_custom_op_symbolic_registry.py (L68)` when exporting `aten::cross_entropy_loss` assumed this, and set the output to be fp16 either. So this is the reason we have the problem. #### Possible Fixes 1. Enhance `SoftmaxCrossEntropyLossInternal` to support different types of input and output. 2. Check the input and output when exporting, add the input case explicitly if there is type promotion from input to output. This PR used the 2nd approach. We can start 1st approach when needed later. TODO: revisit all other exporter functions, add the checks, etc. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->		2022-11-22 15:08:30 +08:00
.config	Update tsaoptions.json: update the email alias (#13448 )	2022-10-26 15:56:16 -07:00
.devcontainer	Remove two lines in the Dockerfile for Github Codespace (#12278 )	2022-07-21 20:52:17 -07:00
.gdn	Update compliance tasks in python packaging pipeline and fix some compile warnings (#8471 )	2021-07-30 17:16:37 -07:00
.github	Convert label config to one line regexes (#13702 )	2022-11-19 11:38:29 -08:00
.pipelines	Remove the cmake option: onnxruntime_DEV_MODE (#13573 )	2022-11-07 09:06:28 -08:00
.vscode	cpplint & Eager mode: refactor and add comments to empty_* functions, general lint cleanup in ort_aten (#12238 )	2022-07-20 11:47:57 -04:00
cgmanifests	Update protobuf-java to version 3.21.7 (#13630 )	2022-11-17 15:04:42 -08:00
cmake	Remove SafeInt dependency from Objective-C API. (#13698 )	2022-11-18 17:06:12 -08:00
csharp	Patch Protobuf and ONNX's cmake files and enforce BinSkim check (#13694 )	2022-11-18 10:09:47 -08:00
dockerfiles	Upgrade cmake version to 3.24 (#13569 )	2022-11-04 22:58:51 -07:00
docs	Improve fp16 performance of GPT-2's logits MatMul while using BeamSearch (#13686 )	2022-11-18 18:50:19 -08:00
include/onnxruntime/core	Allow CUDA EP enable or disable TunableOp via session options and environment variable (#13601 )	2022-11-15 14:43:54 +08:00
java	Update protobuf-java to version 3.21.7 (#13630 )	2022-11-17 15:04:42 -08:00
js	[js] [deps] upgrade @xmldom/xmldom@0.7.9 (#13705 )	2022-11-21 17:01:42 -08:00
objectivec	Remove SafeInt dependency from Objective-C API. (#13698 )	2022-11-18 17:06:12 -08:00
onnxruntime	[ROCm] Add GemmFastGelu TunableOp (#13589 )	2022-11-22 12:58:01 +08:00
orttraining	Make HF converge with lighting native amp (#13616 )	2022-11-22 15:08:30 +08:00
package/rpm	Bumping up version number to 1.14.0 on main branch (#13401 )	2022-10-21 19:16:44 -04:00
samples	Format all python files under onnxruntime with black and isort (#11324 )	2022-04-26 09:35:16 -07:00
tools	Add '-DCMAKE_OSX_ARCHITECTURES=x86_64;arm64' when build protobuf from source on MacOS (#13720 )	2022-11-21 21:59:34 -08:00
winml	Fix WinML Test Case: create LearningModelBinding for every testcase (#13587 )	2022-11-09 11:20:48 +08:00
.clang-format
.clang-tidy	Create clang-tidy CI (#12653 )	2022-09-30 08:05:38 -07:00
.dockerignore	Update dockerfiles (#5929 )	2020-11-25 15:38:22 -08:00
.flake8	Remove miscellaneous nuphar configs (#13070 )	2022-09-26 13:41:28 -07:00
.gitattributes
.gitignore	Ignore settings.json in git (#12988 )	2022-09-19 12:05:43 -07:00
.gitmodules	ignore dirty state of submodule XNNPACK (#13648 )	2022-11-15 00:38:46 -08:00
build.amd64.1411.bat
build.bat
build.sh	Add iOS test pipeline and a sample app. (#5298 )	2020-09-29 13:53:11 -07:00
CITATION.cff	Fix CITATION.cff and add automatic validation of your citation metadata (#10478 )	2022-04-13 10:03:52 -07:00
CODEOWNERS	Add cgmanifest file in codeowner list (#13042 )	2022-09-22 18:58:01 -07:00
CONTRIBUTING.md	minor improvements to CONTRIBUTING doc (#11080 )	2022-04-12 15:22:34 -07:00
lgtm.yml	Fix lgtm C++ error (#13613 )	2022-11-10 10:06:22 -08:00
LICENSE	Remove year from license (#6658 )	2021-02-12 00:25:56 -08:00
NuGet.config	Delete nuget extra configs (#6477 )	2021-01-27 20:25:45 -08:00
ort.wprp
ORT_icon_for_light_bg.png	Update nuget icon (#10672 )	2022-03-01 09:11:03 -08:00
packages.config	Update DML 1.9.0 to 1.9.1 (#12966 )	2022-09-15 10:54:22 -07:00
pyproject.toml	Update pylint config to include valid short names (#13631 )	2022-11-14 10:00:25 -08:00
README.md	Remove miscellaneous nuphar configs (#13070 )	2022-09-26 13:41:28 -07:00
requirements-dev.txt	Introduce parameterized as a dev dependency (#11364 )	2022-04-26 17:24:39 -07:00
requirements-doc.txt	Add auto doc gen for ORTModule API during CI build (#7046 )	2021-03-22 10:20:33 -07:00
requirements-training.txt	Change protobuf pin in training requirements (#13596 )	2022-11-09 09:37:41 -08:00
requirements.txt.in	Add additional python requirements (#11522 )	2022-05-20 16:16:18 -07:00
SECURITY.md	Microsoft mandatory file (#11619 )	2022-05-25 13:56:10 -07:00
setup.py	Enable ORT in TorchDynamo (#13259 )	2022-11-01 11:19:29 -07:00
ThirdPartyNotices.txt	Delete CUB (#13534 )	2022-11-02 13:06:22 -07:00
VERSION_NUMBER	Bumping up version number to 1.14.0 on main branch (#13401 )	2022-10-21 19:16:44 -04:00

README.md

ONNX Runtime is a cross-platform inference and training machine-learning accelerator.

ONNX Runtime inference can enable faster customer experiences and lower costs, supporting models from deep learning frameworks such as PyTorch and TensorFlow/Keras as well as classical machine learning libraries such as scikit-learn, LightGBM, XGBoost, etc. ONNX Runtime is compatible with different hardware, drivers, and operating systems, and provides optimal performance by leveraging hardware accelerators where applicable alongside graph optimizations and transforms. Learn more →

ONNX Runtime training can accelerate the model training time on multi-node NVIDIA GPUs for transformer models with a one-line addition for existing PyTorch training scripts. Learn more →

Get Started

General Information: onnxruntime.ai

Usage documention and tutorials: onnxruntime.ai/docs

Companion sample repositories:

ONNX Runtime Inferencing: microsoft/onnxruntime-inference-examples
ONNX Runtime Training: microsoft/onnxruntime-training-examples

Build Pipeline Status

System	CPU	GPU	EPs
Windows
Linux
Mac
Android
iOS
WebAssembly

Data/Telemetry

Windows distributions of this project may collect usage data and send it to Microsoft to help improve our products and services. See the privacy statement for more details.

Contributions and Feedback

We welcome contributions! Please see the contribution guidelines.

For feature requests or bug reports, please file a GitHub Issue.

For general discussion or questions, please use GitHub Discussions.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

License

This project is licensed under the MIT License.