Scott McKay 9372e9a0a3 Support >2GB of Tensor data in training checkpoint (#20077) ... ### Description <!-- Describe your changes. --> Add ability to store initializer data in an external file. Update training checkpoint code to use external file if data > ~2GB. I don't see a way for the flatbuffers 64-bit offsets to be used, as they don't support storing 'table' types with 64-bit offsets (and our Tensor is a 'table' type not a simple struct). 0cfb7eb80b/tests/64bit/test_64bit.fbs (L38-L39) Allowing a Tensor to have its raw_data in an external file should hopefully work with the least friction. As it's an extra field it's backwards compatible. Please feel free to suggest alternative approaches. Side note: the diffs in the generated *.fbs.h files are unexpectedly large. Maybe they weren't re-generated when the new flatbuffers version was checked in. I updated by running: `python .\compile_schema.py -f <build output dir>\_deps\flatbuffers-build\Debug\flatc.exe` from onnxruntime\core\flatbuffers\schema which I thought was the correct way but maybe that's out of date. I think you can ignore all the diffs in the generated files and just worry about the changes to the .fbs files in onnxruntime/core/flatbuffers/schema. Basically start at the bottom of the files changed and work up as all the 'real' diffs are there. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> --------- Co-authored-by: carzh <wolfivyaura@gmail.com>		2024-04-22 15:17:43 -07:00
..
external	upgrade emsdk to 3.1.57 (#20295)	2024-04-19 23:05:18 -07:00
patches	Add patch for ONNX 1.16.0 shape inference bug (#20316)	2024-04-17 10:23:22 -07:00
tensorboard
adjust_global_compile_flags.cmake	Support xcframework for mac catalyst builds. (#19534)	2024-03-20 10:55:19 -07:00
arm64x.cmake	Build onnxruntime.dll as arm64x (#18633)	2023-12-06 16:49:00 -08:00
CMakeLists.txt	OpenVINO EP Rel 1.18 Changes (#20337)	2024-04-19 00:31:38 -07:00
CMakeSettings.json
codeconv.runsettings
deps.txt	Integration with ONNX 1.16.0 (#19745)	2024-04-12 09:46:49 -07:00
deps_update_and_upload.py	Update google benchmark to 1.8.3. (#19734)	2024-03-01 11:01:58 -08:00
EnableVisualStudioCodeAnalysis.props
gdk_toolchain.cmake
Info.plist.in
libonnxruntime.pc.cmake.in
linux_arm32_crosscompile_toolchain.cmake	Add a build validation for Linux ARM64 cross-compile (#18200)	2023-11-08 13:03:18 -08:00
linux_arm64_crosscompile_toolchain.cmake	Add a build validation for Linux ARM64 cross-compile (#18200)	2023-11-08 13:03:18 -08:00
maccatalyst_prepare_objects_for_prelink.py	Support xcframework for mac catalyst builds. (#19534)	2024-03-20 10:55:19 -07:00
nuget_helpers.cmake
onnxruntime.cmake	Support xcframework for mac catalyst builds. (#19534)	2024-03-20 10:55:19 -07:00
onnxruntime_codegen_tvm.cmake
onnxruntime_common.cmake	Fix build errors from date/date.h C++20 compatibility (#20139)	2024-04-02 22:10:25 -07:00
onnxruntime_compile_triton_kernel.cmake
onnxruntime_config.h.in	Enabling c++ 20 in MacOS build (#16187)	2023-09-26 11:27:02 -07:00
onnxruntime_csharp.cmake
onnxruntime_flatbuffers.cmake
onnxruntime_framework.cmake
onnxruntime_framework.natvis
onnxruntime_fuzz_test.cmake
onnxruntime_graph.cmake	[Apple framework] Fix minimal build with training enabled. (#19858)	2024-03-12 11:33:30 -07:00
onnxruntime_ios.toolchain.cmake
onnxruntime_java.cmake
onnxruntime_java_unittests.cmake
onnxruntime_kernel_explorer.cmake
onnxruntime_language_interop_ops.cmake
onnxruntime_mlas.cmake	Add vectorized AVX512F kernel for ReduceMaximumF32Kernel (#20268)	2024-04-16 13:52:43 -07:00
onnxruntime_nodejs.cmake	Support building Windows CUDA with Ninja (#20176)	2024-04-03 11:19:31 +08:00
onnxruntime_objectivec.cmake
onnxruntime_opschema_lib.cmake
onnxruntime_optimizer.cmake	[ROCm] Fix hipify error: fast_divmod.h: No such file or directory (#19060)	2024-01-10 14:49:19 +08:00
onnxruntime_providers.cmake	Add initial support for CoreML ML Program to the CoreML EP. (#19347)	2024-02-15 08:46:03 +10:00
onnxruntime_providers_acl.cmake	Split onnxruntime_providers.cmake to multiple (#17853)	2023-10-09 20:33:44 -07:00
onnxruntime_providers_armnn.cmake	Split onnxruntime_providers.cmake to multiple (#17853)	2023-10-09 20:33:44 -07:00
onnxruntime_providers_azure.cmake	Split onnxruntime_providers.cmake to multiple (#17853)	2023-10-09 20:33:44 -07:00
onnxruntime_providers_cann.cmake	Split onnxruntime_providers.cmake to multiple (#17853)	2023-10-09 20:33:44 -07:00
onnxruntime_providers_coreml.cmake	Make partitioning utils QDQ aware so it does not break up QDQ node units (#19723)	2024-03-12 10:55:49 +10:00
onnxruntime_providers_cpu.cmake	Revert "Revert NeuralSpeed code for x64 MatMulNBits (#19382)" (#19474)	2024-02-09 09:24:54 -08:00
onnxruntime_providers_cuda.cmake	Enable CUDA EP unit testing on Windows (#20039)	2024-03-27 13:32:36 -07:00
onnxruntime_providers_dml.cmake	Delay load dxcore.dll in addition to ext-ms-win-dxcore-l1-1-0.dll (#18913)	2023-12-26 12:33:42 -08:00
onnxruntime_providers_dnnl.cmake	Split onnxruntime_providers.cmake to multiple (#17853)	2023-10-09 20:33:44 -07:00
onnxruntime_providers_js.cmake	Split onnxruntime_providers.cmake to multiple (#17853)	2023-10-09 20:33:44 -07:00
onnxruntime_providers_migraphx.cmake	CUDA EP vs ROCM EP hipify audit (#17776)	2023-10-13 10:13:53 +08:00
onnxruntime_providers_nnapi.cmake	Make partitioning utils QDQ aware so it does not break up QDQ node units (#19723)	2024-03-12 10:55:49 +10:00
onnxruntime_providers_openvino.cmake	Ort openvino npu 1.17 master (#19966)	2024-03-21 18:44:00 -07:00
onnxruntime_providers_qnn.cmake	Make partitioning utils QDQ aware so it does not break up QDQ node units (#19723)	2024-03-12 10:55:49 +10:00
onnxruntime_providers_rknpu.cmake	Split onnxruntime_providers.cmake to multiple (#17853)	2023-10-09 20:33:44 -07:00
onnxruntime_providers_rocm.cmake	CUDA EP vs ROCM EP hipify audit (#17776)	2023-10-13 10:13:53 +08:00
onnxruntime_providers_tensorrt.cmake	Use CMake's find package for CUDA libs (#19673)	2024-02-27 11:26:48 -08:00
onnxruntime_providers_tvm.cmake	Split onnxruntime_providers.cmake to multiple (#17853)	2023-10-09 20:33:44 -07:00
onnxruntime_providers_vitisai.cmake	[VitisAI] Refactor the VAIEP to use MSFT's standalone API (#19058)	2024-01-31 21:08:26 -08:00
onnxruntime_providers_webnn.cmake	Split onnxruntime_providers.cmake to multiple (#17853)	2023-10-09 20:33:44 -07:00
onnxruntime_providers_xnnpack.cmake	Make partitioning utils QDQ aware so it does not break up QDQ node units (#19723)	2024-03-12 10:55:49 +10:00
onnxruntime_pyop.cmake
onnxruntime_python.cmake	Introducing ORTPipelineModule - DeepSpeed Parallel Pipeline Support. (#20287)	2024-04-18 11:30:15 -07:00
onnxruntime_rocm_hipify.cmake	add QMoE (#20108)	2024-03-29 10:24:19 -07:00
onnxruntime_session.cmake
onnxruntime_snpe_provider.cmake
onnxruntime_training.cmake
onnxruntime_unittests.cmake	Support >2GB of Tensor data in training checkpoint (#20077)	2024-04-22 15:17:43 -07:00
onnxruntime_util.cmake
onnxruntime_webassembly.cmake	upgrade emsdk to 3.1.57 (#20295)	2024-04-19 23:05:18 -07:00
precompiled_header.cmake
riscv64.toolchain.cmake	Enable RISC-V 64-bit Cross-Compiling Support for ONNX Runtime on Linux (#19238)	2024-01-24 16:27:05 -08:00
Sdl.ruleset
set_winapi_family_desktop.h
target_delayload.cmake
uwp_stubs.h
wcos_rules_override.cmake	Stop using apiset in OneCore build: use onecoreuap.lib instead of onecoreuap_apiset.lib (#19632)	2024-02-23 22:31:57 -08:00
winml.cmake	[CP] Fix for xfgcheck and Fix WAI ARM64 build (#19634) (#19644)	2024-03-13 17:54:06 -07:00
winml_cppwinrt.cmake
winml_sdk_helpers.cmake
winml_unittests.cmake	Update C/C++ dependencies: abseil, date, nsync, googletest, wil, mp11, cpuinfo and safeint (#15470)	2023-09-08 13:35:04 -07:00