pengwa 5d8ce817cb Fix simplified layer norm fusion for training (#14866) ... ### Fix simplified layer norm fusion for training Co-author with @prathikr. Fix bug identified by @prathikr. https://github.com/microsoft/onnxruntime/issues/14822. Running T5 model enabling deepspeed, we see simplified layer norm is not fused because the device check did not pass b7fde84341/onnxruntime/core/optimizer/layer_norm_fusion.cc (L568). Since during pretraining optimization pass, there is no device placement, so the device check not fulfilled is expected. On the other hand, the device check is still valid to avoid simplified layer norm fusion works correctly for CPU runs. As a mitigation, added a flag to indicate whether the fusion is triggered by pre-training optimization or not. There is a risk though, when we run ORTModule training with CPU EP, but I feel the risk can be much reduced if we check CUDA/ROCM is enabled for the build. ``` CUDA_VISIBLE_DEVICES=0 python examples/onnxruntime/training/summarization/run_summarization.py --model_name_or_path t5-small --do_train --dataset_name cnn_dailymail --dataset_config "3.0.0" --source_prefix "summarize: " --predict_with_generate --overwrite_output_dir --output_dir /bert_ort/pengwa/output --fp16 --max_steps 1 --logging_steps 1 --deepspeed aml_ds_config_zero_1.json ``` ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->		2023-03-07 13:59:20 -08:00
..
contrib_ops	Implement AllToAll collective op	2023-03-07 10:17:07 -08:00
core	Fix simplified layer norm fusion for training (#14866)	2023-03-07 13:59:20 -08:00
python	enable pybind for qnn ep (#14897)	2023-03-03 07:26:53 -08:00
test	Fix tree implementation when left, right node have lower index (#14839)	2023-03-07 19:47:12 +01:00
tool/etw
wasm	[js/web] support flag 'optimizedModelFilePath' in session options (#14355)	2023-02-24 15:50:15 -08:00
__init__.py	Add GetVersionSting API for C++, C# and Python (#14873)	2023-03-02 17:11:07 -08:00
ReformatSource.ps1
ReformatSourcePython.bat
VSCodeCoverage.runsettings