saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-08 17:17:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/orttraining/orttraining/python
History
zhijiang 8fadc6c913
Zhijxu/cleanup cached tensors when oom (#19306) 
in pytorch, when oom happens at bp, user could decrease the batch size
and rerun it without restarting the process.

while in ORT, the intermediate tensors are kept even OOM, so decrease
batch size still fail.


this is torch run, we can see after oom failure, torch will release
tensor before next step

![image](https://github.com/microsoft/onnxruntime/assets/43435212/92b8a2e3-454b-448a-a223-17cb91d463c2)

this is from ort, we can see ort not release its tensors after OOM
failure.

![image](https://github.com/microsoft/onnxruntime/assets/43435212/bb6a3882-8e14-4f37-8079-e7f70fc2546b)

ort with the PR, we can see memory is released, **the 4GB memory is not
own by ort, and will be released by torch at the end**.

![image](https://github.com/microsoft/onnxruntime/assets/43435212/7f39d711-4e36-47d5-aecf-3805433a6d01)
2024-02-21 10:41:42 +08:00
..
training Zhijxu/cleanup cached tensors when oom (#19306) 2024-02-21 10:41:42 +08:00
orttraining_pybind_common.h Re-work global objects dependancies in pybind layer. (#14941) 2023-03-10 13:55:31 -08:00
orttraining_pybind_state.cc Introduce a Nominal Checkpoint for On-Device Training (#19232) 2024-01-30 22:11:25 -08:00
orttraining_python_module.cc [CUDA][ROCm][Training] Fix cuda/rocm provider info hash (#19398) 2024-02-05 14:35:57 -08:00
orttraining_python_module_eager.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
pt_patch.py Adopt linrtunner as the linting tool - take 2 (#15085) 2023-03-24 15:29:03 -07:00