pengwa 8a98874e7e Flash attention recompute (#20603) ... ### Flash attn recompute 1. Allow PythonOp(FlashAttn) can be recomputed correctly. 45879ff5c2 2. Use JSON to pass the selected-to-recompute subgraphs. 3c374da678 #### Better Memory Efficiency Customer model can run both PyTorch SPDA and Flash Attn, this PR make it possible to let the Flash Attn path work with ORTModule layerwise recompute. The peak drop from 45.xGB to 32.xGB if we only compare the layers (not including other pieces, BTW there are few more optimization targeting other pieces as well later). #### Better Perf Using Flash ATTN bring additionally 16% end to end time reduction, with highly aligned loss curve.  #### Use JSON File to pass Recompute Plans To overcome the limitation of max length of the strings defined in session options. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. -->		2024-05-21 13:38:19 +08:00
..
c_cxx
execution_providers/images
images
python	Bump up version in main from 1.18.0 to 1.19.0 (#20489)	2024-04-29 20:21:41 -07:00
ABI_Dev_Notes.md
Android_testing.md
C_API_Guidelines.md
cmake_guideline.md
Coding_Conventions_and_Standards.md
ContribOperators.md	MatMulNBits + Add fusion (#20587)	2024-05-16 11:00:59 -07:00
FAQ.md
How_To_Update_ONNX_Dev_Notes.md	Integration with ONNX 1.16.0 (#19745)	2024-04-12 09:46:49 -07:00
Memory_Optimizer.md	Flash attention recompute (#20603)	2024-05-21 13:38:19 +08:00
Model_Test.md	Update docs/Model_Test.md (#11466)	2024-05-15 11:33:11 -07:00
NotesOnThreading.md
ONNX_Runtime_Server_Usage.md
onnxruntime_dependencies.dot
onnxruntime_dependencies.png
onnxruntime_extensions.md
OperatorKernels.md	MatMulNBits + Add fusion (#20587)	2024-05-16 11:00:59 -07:00
ORT_Format_Update_in_1.13.md
ORT_Use_Trtion_Kernel.md
ORTMobilePackageOperatorTypeSupport.md
ORTModule_Convergence_Notes.md	Fix and enable few ORTModule Unit Tests (#19847)	2024-03-12 10:49:19 +08:00
ORTModule_ModuleWithLoss_Wrapper.md
ORTModule_PythonOp_Notes.md
ORTModule_Training_Guidelines.md	Generalize label input sparsity check and refactor (#20636)	2024-05-10 21:55:43 +08:00
PR_Guidelines.md
Privacy.md	[C# and Python APIs] Expose knobs to enable/disable platform telemetry collection (#5481)	2020-10-21 10:32:13 -07:00
Python_Dev_Notes.md
Reduced_Operator_Kernel_build.md
ReleaseManagement.md
Roadmap.md
Server.md
TVM_EP.md
Versioning.md
WinML_principles.md