mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-23 19:32:23 +00:00
14d7872ce9
### Description It always has been out of memory in training CUDA 12.2 packaging pipeline https://dev.azure.com/aiinfra/Lotus/_build?definitionId=1308&_a=summary since the PR #19910 I tried other CPU agents for example, D64as_v5(256G memory) and D32as_v4(128G memory and 256 G SSD temp storage), which are still out of memory like the below image  But it works on T4, though T4 only has 4 vCPUs, 28G memory and 180G temp storage, and it takes much more time. ### Motivation and Context Restore CUDA 12.2 training packaging pipeline first. More time is needed to investigate the root cause ### Other Clues. These 2 compilation steps take nearly 6 minutes with Cuda 12.2 on T4 And it runs out of memory on CPU machine. @ajindal1 cuda12.2 on T4 ``` 2024-03-14T05:39:08.7726865Z [ 90%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/onnxruntime_src/onnxruntime/contrib_ops/cuda/bert/flash_attention/flash_fwd_split_hdim32_fp16_sm80.cu.o 2024-03-14T05:45:01.3223393Z [ 90%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/onnxruntime_src/onnxruntime/contrib_ops/cuda/bert/flash_attention/flash_fwd_split_hdim64_bf16_sm80.cu.o 2024-03-14T05:46:07.9218003Z [ 90%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/onnxruntime_src/onnxruntime/contrib_ops/cuda/bert/flash_attention/flash_fwd_split_hdim96_fp16_sm80.cu.o 2024-03-14T05:52:59.2387051Z [ 90%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/onnxruntime_src/onnxruntime/contrib_ops/cuda/bert/group_query_attention_impl.cu.o ``` But they could be finished in about one minute with Cuda 11.8 on CPU ``` cuda11.8 on CPU 2024-04-09T11:34:35.0849836Z [ 90%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/onnxruntime_src/onnxruntime/contrib_ops/cuda/bert/flash_attention/flash_fwd_split_hdim32_fp16_sm80.cu.o 2024-04-09T11:35:53.6648154Z [ 90%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/onnxruntime_src/onnxruntime/contrib_ops/cuda/bert/flash_attention/flash_fwd_split_hdim64_bf16_sm80.cu.o cuda11.8 on GPU 024-03-13T12:16:33.4102477Z [ 90%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/onnxruntime_src/onnxruntime/contrib_ops/cuda/bert/flash_attention/flash_fwd_split_hdim32_fp16_sm80.cu.o 2024-03-13T12:19:58.8268272Z [ 90%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/onnxruntime_src/onnxruntime/contrib_ops/cuda/bert/flash_attention/flash_fwd_split_hdim64_bf16_sm80.cu.o ``` |
||
|---|---|---|
| .. | ||
| jobs | ||
| stages | ||
| android-binary-size-check-stage.yml | ||
| android-dump-logs-from-steps.yml | ||
| android-java-api-aar-test.yml | ||
| android-java-api-aar.yml | ||
| build-linux-wasm-step.yml | ||
| c-api-artifacts-package-and-publish-steps-posix.yml | ||
| c-api-artifacts-package-and-publish-steps-windows.yml | ||
| c-api-cpu.yml | ||
| c-api-linux-cpu.yml | ||
| check-cache-stats.yml | ||
| check_test_result.yml | ||
| clean-agent-build-directory-step.yml | ||
| common-variables.yml | ||
| compliance.yml | ||
| component-governance-component-detection-steps.yml | ||
| download-deps.yml | ||
| esrp_nuget.yml | ||
| explicitly-defined-final-tasks.yml | ||
| final-jar-testing.yml | ||
| flex-downloadPipelineArtifact.yml | ||
| get-docker-image-steps.yml | ||
| install-appcenter.yml | ||
| java-api-artifacts-package-and-publish-steps-posix.yml | ||
| linux-build-step-with-cache.yml | ||
| linux-ci.yml | ||
| linux-cpu-packaging-pipeline.yml | ||
| linux-gpu-tensorrt-packaging-pipeline.yml | ||
| linux-wasm-ci.yml | ||
| linux-web-init-and-check.yml | ||
| mac-build-step-with-cache.yml | ||
| mac-cpu-packaging-pipeline.yml | ||
| mac-cpu-packaging-steps.yml | ||
| mac-cpu-packing-jobs.yml | ||
| mac-esrp-dylib.yml | ||
| make_java_win_binaries.yml | ||
| nodejs-artifacts-package-and-publish-steps-posix.yml | ||
| nodejs-artifacts-package-and-publish-steps-windows.yml | ||
| ondevice-training-cpu-packaging-pipeline.yml | ||
| orttraining-linux-gpu-test-ci-pipeline.yml | ||
| py-linux-gpu.yml | ||
| py-linux.yml | ||
| py-package-smoking-test.yml | ||
| py-packaging-linux-test-cpu.yml | ||
| py-packaging-linux-test-cuda.yml | ||
| py-packaging-selectable-stage.yml | ||
| py-packaging-stage.yml | ||
| py-packaging-training-cuda-stage-steps.yml | ||
| py-packaging-training-cuda-stage.yml | ||
| py-win-arm64-qnn.yml | ||
| py-win-gpu.yml | ||
| py-win-x64-qnn.yml | ||
| react-native-ci.yml | ||
| rocm.yml | ||
| run-docker-build-steps.yml | ||
| set-nightly-build-option-variable-step.yml | ||
| set-python-manylinux-variables-step.yml | ||
| set-version-number-variables-step.yml | ||
| telemetry-steps.yml | ||
| upload-code-coverage-data.yml | ||
| use-android-emulator.yml | ||
| use-android-ndk.yml | ||
| use-xcode-version.yml | ||
| validate-package.yml | ||
| web-browserstack-ci.yml | ||
| web-ci.yml | ||
| win-ci.yml | ||
| win-esrp-dll.yml | ||
| win-wasm-ci.yml | ||
| win-web-ci.yml | ||
| win-web-multi-browsers.yml | ||
| with-container-registry-steps.yml | ||