pengwa 735a32fee1 Introduce memory observer for ORTModule (#16213) ... ### Introduce memory observer for ORTModule To analyze memory usage for ORTModule training, we need collect per-iteration memory footprint in different stages (pre-forward, post-forward, pre-backward, and post-backward). Currently we only collect the data using torch.cuda APIs. The next step is, we could collect the detailed stashed activation list and its percentage within ORT backend, which is beyond this PR. Sample as below: ``` 0/8] step 0 memory (MiB) | phase: pre_forward | allocated: 1866 | max allocated: 1866 | cached: 1874 | max cached: 1874 | inactive: 8 | max inactive: 8 [0/8] step 0 memory (MiB) | phase: post_forward | allocated: 23277 | max allocated: 26215 | cached: 26406 | max cached: 26406 | inactive: 193 | max inactive: 405 [0/8] step 0 memory (MiB) | phase: pre_backward | allocated: 23277 | max allocated: 26215 | cached: 26406 | max cached: 26406 | inactive: 193 | max inactive: 405 [0/8] step 0 memory (MiB) | phase: post_backward | allocated: 2932 | max allocated: 26215 | cached: 26406 | max cached: 26406 | inactive: 6158 | max inactive: 6158 0%|█ | 1/200 [00:26<1:26:18, 26.02s/it] [0/8] step 1 memory (MiB) | phase: pre_forward | allocated: 2356 | max allocated: 26215 | cached: 26406 | max cached: 26406 | inactive: 2454 | max inactive: 6165 [0/8] step 1 memory (MiB) | phase: post_forward | allocated: 23767 | max allocated: 26705 | cached: 29342 | max cached: 29342 | inactive: 2639 | max inactive: 6165 [0/8] step 1 memory (MiB) | phase: pre_backward | allocated: 23767 | max allocated: 26705 | cached: 29342 | max cached: 29342 | inactive: 2639 | max inactive: 6165 [0/8] step 1 memory (MiB) | phase: post_backward | allocated: 3422 | max allocated: 26705 | cached: 29342 | max cached: 29342 | inactive: 5284 | max inactive: 6165 1%|██ | 2/200 [00:26<36:47, 11.15s/it] [0/8] step 2 memory (MiB) | phase: pre_forward | allocated: 2356 | max allocated: 26705 | cached: 29342 | max cached: 29342 | inactive: 2454 | max inactive: 6165 [0/8] step 2 memory (MiB) | phase: post_forward | allocated: 23767 | max allocated: 26705 | cached: 29342 | max cached: 29342 | inactive: 2639 | max inactive: 6165 [0/8] step 2 memory (MiB) | phase: pre_backward | allocated: 23767 | max allocated: 26705 | cached: 29342 | max cached: 29342 | inactive: 2639 | max inactive: 6165 [0/8] step 2 memory (MiB) | phase: post_backward | allocated: 3422 | max allocated: 26705 | cached: 29342 | max cached: 29342 | inactive: 5284 | max inactive: 6165 ```		2023-06-15 15:45:36 +08:00
..
c_cxx	Training Documentation (#15612)	2023-04-25 11:44:12 -07:00
execution_providers/images	Remove docs that have been migrated to https://onnxruntime.ai/docs (#6225)	2021-02-05 18:09:27 -08:00
images	API Documentation (#8948)	2021-09-09 22:04:51 -07:00
python	Add C# API documentation for training (and some other changes) (#15935)	2023-05-16 03:15:24 -07:00
ABI_Dev_Notes.md	skip windows GPU check if changes only in doc (#13248)	2022-10-11 13:51:44 +08:00
Android_testing.md	Removed BUILD.md from master as source now lives in gh-pages (#6709)	2021-02-19 11:34:21 -08:00
C_API_Guidelines.md	Replace 'master' branch ref to 'main' in the code (#12547)	2022-08-22 10:48:12 -07:00
cmake_guideline.md	fix some typo in docs (#13212)	2022-10-07 15:58:18 -07:00
Coding_Conventions_and_Standards.md	Enable RUFF as a formatter (#15699)	2023-04-26 14:04:07 -07:00
ContribOperators.md	optimization for whisper model with decoder masked multihead attention (#15827)	2023-05-18 15:38:31 -07:00
FAQ.md	Fix typo enviroment => environment (#13195)	2022-10-03 17:02:26 -07:00
How_To_Update_ONNX_Dev_Notes.md	Remove exclusions for ONNX model tests that now pass. (#14337)	2023-01-24 08:04:27 +10:00
Memory_Optimizer.md	Add guidelines for ORTModule (#13553)	2022-11-04 19:42:10 +08:00
Model_Test.md
NotesOnThreading.md	Replace 'master' branch ref to 'main' in the code (#12547)	2022-08-22 10:48:12 -07:00
ONNX_Runtime_Server_Usage.md	Update docs/ONNX_Runtime_Server_Usage.md (#7818)	2021-05-26 16:17:20 -07:00
onnxruntime_dependencies.dot
onnxruntime_dependencies.png
onnxruntime_extensions.md	Fix broken and outdated links in documentation (#14092)	2023-02-23 10:48:04 -08:00
OperatorKernels.md	[DML EP] Register Div with int64 and NonZero with bool (#16276)	2023-06-08 13:49:39 -07:00
ORT_Format_Update_in_1.13.md	Update ORT format v5 change docs to cover limited backwards compatibility in 1.14. (#14413)	2023-01-25 08:23:12 -08:00
ORT_use_trtion_kernel.md	integrate triton into ort (#15862)	2023-05-17 09:35:28 +08:00
ORTMobilePackageOperatorTypeSupport.md	Replace 'master' branch ref to 'main' in the code (#12547)	2022-08-22 10:48:12 -07:00
ORTModule_Convergence_Notes.md	Enhance StatisticsSubscriber (#16098)	2023-06-12 18:32:08 +08:00
ORTModule_Training_Guidelines.md	Introduce memory observer for ORTModule (#16213)	2023-06-15 15:45:36 +08:00
PR_Guidelines.md
Privacy.md	[C# and Python APIs] Expose knobs to enable/disable platform telemetry collection (#5481)	2020-10-21 10:32:13 -07:00
Python_Dev_Notes.md
Reduced_Operator_Kernel_build.md	replace 'master' branch ref to 'main' for onnx repo (#12678)	2022-08-30 13:41:42 -07:00
ReleaseManagement.md
Roadmap.md	Replace 'master' branch ref to 'main' in the code (#12547)	2022-08-22 10:48:12 -07:00
Server.md	Update documentation for contributing a PR and add deprecation notices for PyOp and ORT server. (#6172)	2020-12-18 02:00:42 -08:00
TVM_EP.md	Update python 3.11 and remove 3.7 for Linux (#15214)	2023-03-27 14:46:30 -07:00
Versioning.md	replace 'master' branch ref to 'main' for onnx repo (#12678)	2022-08-30 13:41:42 -07:00
WinML_principles.md	Replace 'master' branch ref to 'main' in the code (#12547)	2022-08-22 10:48:12 -07:00