saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-27 20:02:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/onnxruntime/core/framework
History
Hector Li 5daeb5e0b0
enable model with external data be loaded from memory buffer (#19089) 
### Description
Background:
User save large model with initializer data in external file. e.g:
onnx.save_model(onnx_model, "path/to/save/the/model.onnx", save_as_external_data=True, all_tensors_to_one_file=True,
location="filename", size_threshold=1024).
In that case, Ort loads the model, get the external initializer information (external file name, offset, length) and use the model path to find the external file, and locate to the tensor data via the offset and length.
But it won't work if user load the model from memory, since Ort lost track of the model path.

This PR adds API/session option to let user provide a table with external initializer file name as the key, the pointer to the loaded external file in memory and the buffer length as value. So that

1. user can load the model from memory buffer with external initializers in memory buffer too.
2. the initializers can be shared across sessions, for different EPs.
3. user can load the file in any way they want, e.g mmap.

Internally, 
1. at session creation time, Ort goes through the external initializers in the graph, gets the file name, offset, data length of the external initializers from Tensorproto .
2. With the file name, Ort get the file in memory buffer and buffer length from the table user provided.
4. Ort locates the tensor buffer from file in memory buffer (user provided) using the offset and data length (from Tensorproto ).
5. Ort creates the Tensor and replace the existing Tensor in the graph.


### Motivation and Context
https://github.com/onnx/onnx/blob/main/docs/ExternalData.md
For a model with external data, the Tensorproto may have initializer data in a separate file. The external file location is set via the file path relative to the model path. With the API to load model from memory buffer, it lost track of the
model path. So it causes error if the model has external data. By adding a session option to set the external data buffer, Ort can find the external data correctly if model loaded from memory buffer.
2024-04-17 19:01:01 -07:00
..
allocation_planner.cc Regarding copy inputs before inference, flush the stream which copies the input only if the input is consumed by the ops from different streams (#19970) 2024-03-26 13:57:25 -07:00
allocation_planner.h ExecutionProvider API refactor - replace OrtMemoryInfo with OrtDevice (#15618) 2023-05-01 10:06:00 -07:00
allocator.cc Throw if unique_ptr or array allocation fails due to SafeInt overflow (#18941) 2024-01-03 07:57:51 +10:00
allocator_stats.h
allocator_utils.cc remove AllocatorMgr class (#16509) 2023-06-28 15:43:19 -07:00
allocator_utils.h remove AllocatorMgr class (#16509) 2023-06-28 15:43:19 -07:00
arena_extend_strategy.h
bfc_arena.cc New configuration to limit the arena extension (#15983) 2023-05-25 02:19:07 -07:00
bfc_arena.h Align bins_space_ storage (#17552) 2024-02-11 19:18:26 -08:00
callback.cc
callback.h
compute_capability.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
config_options.cc Log out ORT session options (#16259) 2023-12-11 17:56:27 -08:00
config_options.h Log out ORT session options (#16259) 2023-12-11 17:56:27 -08:00
copy.cc
copy.h Add support for other data types to Split CPU kernel. (#13900) 2022-12-12 09:29:15 -08:00
customregistry.cc
data_transfer.cc Multi-stream execution support (#13495) 2022-12-15 07:39:29 -08:00
data_transfer.h Multi-stream execution support (#13495) 2022-12-15 07:39:29 -08:00
data_transfer_manager.cc Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
data_transfer_manager.h Multi-stream execution support (#13495) 2022-12-15 07:39:29 -08:00
data_transfer_utils.h Switch GSL to MS GSL 4.0.0 (#13416) 2022-10-29 04:15:20 -07:00
data_types.cc [C#, CPP] Introduce Float16/BFloat16 support and tests for C#, C++ (#16506) 2023-07-14 10:46:52 -07:00
debug_node_inputs_outputs_utils.cc dump tensor statistics (#15761) 2023-08-16 10:53:48 -07:00
debug_node_inputs_outputs_utils.h dump tensor statistics (#15761) 2023-08-16 10:53:48 -07:00
device_stream_collection.cc clean unused parameter in ORT_UNUSED_PARAMETER (#16538) 2023-07-07 13:20:36 -07:00
device_stream_collection.h ExecutionProvider API refactor - move allocator from EP level to SessionState level and indexed by OrtDevice (#15833) 2023-06-19 17:44:45 -07:00
element_type_lists.h Introduce float 8 types (#14731) 2023-05-30 13:25:58 -07:00
empty.cc
endian_utils.cc Remove CUDA 10.2 support (#12541) 2022-08-10 22:46:41 -07:00
endian_utils.h Switch GSL to MS GSL 4.0.0 (#13416) 2022-10-29 04:15:20 -07:00
error_code.cc Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
error_code_helper.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
ex_lib_loader.cc Support loading widechar paths on windows (#14066) 2022-12-30 16:30:11 -08:00
ex_lib_loader.h
execution_frame.cc Zhijxu/cleanup cached tensors when oom (#19306) 2024-02-21 10:41:42 +08:00
execution_frame.h Zhijxu/cleanup cached tensors when oom (#19306) 2024-02-21 10:41:42 +08:00
execution_plan_base.h ExecutionProvider API refactor - replace OrtMemoryInfo with OrtDevice (#15618) 2023-05-01 10:06:00 -07:00
execution_provider.cc ExecutionProvider API refactor - make GenerateMetaDefId a standalone function, decouple it from EP (#18977) 2024-01-26 07:39:08 -08:00
execution_providers.h Add capturestate / rundown ETW support logging for session and provider options (#19397) 2024-02-08 11:28:05 -08:00
execution_steps.cc Refine logging for execution plan print (#19777) 2024-03-14 16:31:32 +08:00
execution_steps.h Refine logging for execution plan print (#19777) 2024-03-14 16:31:32 +08:00
fallback_cpu_capability.cc [ORTModule] ATen Efficient Attention and Triton Flash Attention (#17959) 2023-10-27 10:29:27 +08:00
fallback_cpu_capability.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
feeds_fetches_manager.cc Eliminate memory allocations per recent profiling (#12225) 2022-07-25 14:14:38 -07:00
feeds_fetches_manager.h use OrtCustomOp's new API GetMayInplace in CreateKernelCreateInfo (#20037) 2024-03-28 20:45:37 -07:00
func_kernel.cc
func_kernel.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
fuse_nodes_funcs.cc Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
fuse_nodes_funcs.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
graph_partitioner.cc Multi-partition support for context binary cache feature (#18865) 2024-02-01 15:04:29 -08:00
graph_partitioner.h Fix issue that the generated context cache model inputs/outputs order is not guaranteed (#19195) 2024-01-19 15:16:17 -08:00
iexecutor.h ExecutionProvider API refactor - replace OrtMemoryInfo with OrtDevice (#15618) 2023-05-01 10:06:00 -07:00
kernel_def_builder.cc Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
kernel_lookup.h Switch GSL to MS GSL 4.0.0 (#13416) 2022-10-29 04:15:20 -07:00
kernel_registry.cc Refine cast optimizer for safety (#15658) 2023-04-28 09:32:54 -07:00
kernel_registry_manager.cc Make session configuration options available to kernels via OpKernelInfo (#18897) 2024-01-13 10:02:43 +10:00
kernel_registry_manager.h Switch GSL to MS GSL 4.0.0 (#13416) 2022-10-29 04:15:20 -07:00
kernel_type_str_resolver.cc Introduce float 8 types (#14731) 2023-05-30 13:25:58 -07:00
kernel_type_str_resolver.h [On-Device-Training] Upgrade Flatbuffers to Support 2GB+ Checkpoints. (#19770) 2024-03-14 16:36:24 -07:00
kernel_type_str_resolver_utils.cc Integration with ONNX 1.16.0 (#19745) 2024-04-12 09:46:49 -07:00
kernel_type_str_resolver_utils.h Switch GSL to MS GSL 4.0.0 (#13416) 2022-10-29 04:15:20 -07:00
library_handles.cc Improve custom op library handle cleanup (#14099) 2023-01-04 17:56:29 -08:00
library_handles.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
math.h Switch GSL to MS GSL 4.0.0 (#13416) 2022-10-29 04:15:20 -07:00
mem_buffer.h
mem_pattern.h ExecutionProvider API refactor - replace OrtMemoryInfo with OrtDevice (#15618) 2023-05-01 10:06:00 -07:00
mem_pattern_planner.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
memcpy.cc Multi-stream execution support (#13495) 2022-12-15 07:39:29 -08:00
memcpy.h
memory_info.cc fix memory profile build (#16177) 2023-05-31 16:08:14 -07:00
memory_info.h fix memory profile build (#16177) 2023-05-31 16:08:14 -07:00
mldata_type_utils.cc
mldata_type_utils.h
model_metadef_id_generator.cc ExecutionProvider API refactor - make GenerateMetaDefId a standalone function, decouple it from EP (#18977) 2024-01-26 07:39:08 -08:00
model_metadef_id_generator.h ExecutionProvider API refactor - make GenerateMetaDefId a standalone function, decouple it from EP (#18977) 2024-01-26 07:39:08 -08:00
murmurhash3.cc Add big endian support to murmurhash3 (#12549) 2022-08-11 18:39:39 +10:00
murmurhash3.h
node_index_info.cc Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
node_index_info.h Retry Rework execution frame to reduce memory allocations (#11897) 2022-06-20 10:29:43 -07:00
node_unit.cc [QNN EP] Support per-channel quantized weights (#20154) 2024-04-16 08:45:35 -07:00
node_unit.h [QNN EP] Support per-channel quantized weights (#20154) 2024-04-16 08:45:35 -07:00
onnxruntime_map_type_info.cc Introduce float 8 types (#14731) 2023-05-30 13:25:58 -07:00
onnxruntime_map_type_info.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
onnxruntime_optional_type_info.cc Implement Optional Metadata support and C# test support (#15314) 2023-04-11 09:41:59 -07:00
onnxruntime_optional_type_info.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
onnxruntime_sequence_type_info.cc Implement Optional Metadata support and C# test support (#15314) 2023-04-11 09:41:59 -07:00
onnxruntime_sequence_type_info.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
onnxruntime_typeinfo.cc Introduce float 8 types (#14731) 2023-05-30 13:25:58 -07:00
onnxruntime_typeinfo.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
op_kernel.cc ExecutionProvider API refactor - replace OrtMemoryInfo with OrtDevice (#15618) 2023-05-01 10:06:00 -07:00
op_kernel_context_internal.h Multi-stream execution support (#13495) 2022-12-15 07:39:29 -08:00
op_kernel_info.cc Make session configuration options available to kernels via OpKernelInfo (#18897) 2024-01-13 10:02:43 +10:00
op_kernel_type_control_utils.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
op_node_proto_helper.cc MLAS AArch64 quantized int4 Gemm kernel (#18031) 2023-11-15 09:31:54 -08:00
ort_stl_allocator.h
ort_value_name_idx_map.h Build VS 2022 no Abseil adjustment (#12780) 2022-08-31 11:47:43 -07:00
ort_value_pattern_planner.cc Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
ort_value_pattern_planner.h ExecutionProvider API refactor - replace OrtMemoryInfo with OrtDevice (#15618) 2023-05-01 10:06:00 -07:00
ort_value_tensor_slicer.cc Eliminate memory allocations per recent profiling (#12225) 2022-07-25 14:14:38 -07:00
ort_value_tensor_slicer.h
partial_graph_execution_state.cc Enable conditional optimization automatically (#15885) 2023-05-23 13:08:05 +08:00
partial_graph_execution_state.h Avoid the lock for device stream impact the cpu build (#14131) 2023-01-05 09:01:33 -08:00
prepacked_weights.cc
prepacked_weights.h Refactor prepack buffer code (#16280) 2023-06-08 14:42:02 -07:00
prepacked_weights_container.cc remove AllocatorMgr class (#16509) 2023-06-28 15:43:19 -07:00
prepacked_weights_container.h Revert "Refactor ExecutionFrame and SessionState to reduce memory all… (#11888) 2022-06-17 17:07:21 +08:00
print_tensor_statistics_utils.h dump tensor statistics (#15761) 2023-08-16 10:53:48 -07:00
print_tensor_utils.h Enable -Wshorten-64-to-32 warning if available. (#16524) 2023-07-07 08:11:44 -07:00
program_region.h Multi-stream execution support (#13495) 2022-12-15 07:39:29 -08:00
random_generator.cc
random_generator.h
random_seed.cc Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
random_seed.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
run_options.cc
sequential_execution_plan.h Regarding copy inputs before inference, flush the stream which copies the input only if the input is consumed by the ops from different streams (#19970) 2024-03-26 13:57:25 -07:00
sequential_executor.cc Add support for LRN NHWC OPs (#19866) 2024-03-13 17:52:07 -07:00
sequential_executor.h Fix memory profiler (#14695) 2023-02-23 18:05:21 +08:00
session_options.cc enable model with external data be loaded from memory buffer (#19089) 2024-04-17 19:01:01 -07:00
session_options.h enable model with external data be loaded from memory buffer (#19089) 2024-04-17 19:01:01 -07:00
session_state.cc Add prepacked weights container to subgraphs (#17671) 2023-09-26 12:01:41 -07:00
session_state.h Regarding copy inputs before inference, flush the stream which copies the input only if the input is consumed by the ops from different streams (#19970) 2024-03-26 13:57:25 -07:00
session_state_utils.cc Regarding copy inputs before inference, flush the stream which copies the input only if the input is consumed by the ops from different streams (#19970) 2024-03-26 13:57:25 -07:00
session_state_utils.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
simple_tensor_allocator.cc ExecutionProvider API refactor - replace OrtMemoryInfo with OrtDevice (#15618) 2023-05-01 10:06:00 -07:00
simple_tensor_allocator.h ExecutionProvider API refactor - replace OrtMemoryInfo with OrtDevice (#15618) 2023-05-01 10:06:00 -07:00
sparse_tensor.cc Throw if unique_ptr or array allocation fails due to SafeInt overflow (#18941) 2024-01-03 07:57:51 +10:00
sparse_utils.cc Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
sparse_utils.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
stream_execution_context.cc Enable user to set QNN HTP performance mode for every session run (#19521) 2024-02-22 17:04:59 -08:00
stream_execution_context.h Enable -Wshorten-64-to-32 warning if available. (#16524) 2023-07-07 08:11:44 -07:00
tensor.cc Handle string initializers in constant folding (#17422) 2023-09-27 21:15:58 +10:00
tensor_allocator.cc ExecutionProvider API refactor - replace OrtMemoryInfo with OrtDevice (#15618) 2023-05-01 10:06:00 -07:00
tensor_allocator.h ExecutionProvider API refactor - replace OrtMemoryInfo with OrtDevice (#15618) 2023-05-01 10:06:00 -07:00
tensor_allocator_with_mem_pattern.h ExecutionProvider API refactor - replace OrtMemoryInfo with OrtDevice (#15618) 2023-05-01 10:06:00 -07:00
tensor_external_data_info.cc
tensor_external_data_info.h Support direct usage of ORT format model flatbuffer for initializers (#12465) 2022-08-12 18:31:43 +10:00
tensor_shape.cc Fix typo in the TensorShape (#17813) 2023-12-01 16:48:55 -08:00
tensor_type_and_shape.cc Custom op shape inference API (#17737) 2023-10-13 12:57:42 -07:00
tensor_type_and_shape.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
tensorprotoutils.cc [js] enable external data loading for ort-web (#19087) 2024-01-12 19:24:24 -08:00
tensorprotoutils.h Handle string initializers in constant folding (#17422) 2023-09-27 21:15:58 +10:00
TensorSeq.h Enable Opset11 Sequence Ops on DirectML, and make the CPU implementations agnostic to backend EP (#14442) 2023-02-21 18:08:28 -08:00
transform_layout_functions.h Re-organize the transpose optimization and layout transformation files. (#16246) 2023-07-07 08:24:47 +10:00
transpose_helper.cc Parallelize Transpose (#16854) 2023-08-07 14:25:53 -07:00
transpose_helper.h Parallelize Transpose (#16854) 2023-08-07 14:25:53 -07:00
tunable.h Improve tunable verbose log (#17328) 2023-10-31 13:10:21 +08:00
tuning_context.h remove gsl to make status.h independent from gsl (#17402) 2023-09-13 21:47:43 -07:00
tuning_context_impl.h Run clang-format in CI (#15524) 2023-04-18 09:26:58 -07:00
tuning_results.h Add TuningContext for TunableOp (#14557) 2023-02-10 14:27:43 +08:00
utils.cc Regarding copy inputs before inference, flush the stream which copies the input only if the input is consumed by the ops from different streams (#19970) 2024-03-26 13:57:25 -07:00
utils.h [ORTModule] ATen Efficient Attention and Triton Flash Attention (#17959) 2023-10-27 10:29:27 +08:00