mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-06-28 03:20:58 +00:00
9790e19424
* Fixes from investigating issue running BERT-Squad model with larger batch sizes. When the batch size gets large enough the initial run will be successful (no memory pattern in use) but the second will fail to allocate the memory pattern block. The cause of this failure is that we still have the smaller blocks from the first run allocated, as BFCArena has no logic to free those. This essentially results in 2x the memory being required to run the model. There was inconsistency in BFCArena::Extend which on one path threw an exception if it couldn't do the allocation, and on another just returned false (resulting in Alloc returning a nullptr). Make the behavior consistent by always throwing if BFCArena fails to find a buffer to return. There are a huge number of places in the code where we assume Alloc returns a valid pointer so throwing will result in more correct behavior as a whole. It's also consistent with what happens when CUDA or the standard library fails to allocate memory. Next, update ExecutionFrame to check for this failure and not insert a memory block entry if it happens. With the existing code if BFCArena Alloc returned a nullptr we happily inserted that in the blocks, delaying detection of the failure to when we attempted to use the block in AllocateMLValueTensorSelfOwnBufferHelper. Finally update AllocateMLValueTensorSelfOwnBufferHelper to expect a location may not have a block. A log message will be provided when the block allocation fails so it's not necessary to have more on each individual allocation that would have used the block. Falls through to default behavior of doing a normal allocation. |
||
|---|---|---|
| .. | ||
| cuda | ||
| allocation_planner_test.cc | ||
| allocator_test.cc | ||
| bfc_arena_test.cc | ||
| data_types_test.cc | ||
| distance_test.cc | ||
| dummy_allocator.cc | ||
| dummy_allocator.h | ||
| dummy_provider.cc | ||
| dummy_provider.h | ||
| endian_test.cc | ||
| execution_frame_test.cc | ||
| float_16_test.cc | ||
| inference_session_test.cc | ||
| insert_cast_transformer_test.cc | ||
| kernel_registry_test.cc | ||
| local_kernel_registry_test.cc | ||
| math_test.cc | ||
| mem_pattern_planner_test.cc | ||
| memcpy_transformer_test.cc | ||
| model_builder_utils.h | ||
| opaque_kernels_test.cc | ||
| parallel_executor_test.cc | ||
| random_test.cc | ||
| session_state_test.cc | ||
| shape_inference_test.cc | ||
| sparse_kernels_test.cc | ||
| tensor_test.cc | ||
| tensorutils_test.cc | ||
| test_main.cc | ||
| test_tensor_loader.cc | ||
| test_utils.cc | ||
| test_utils.h | ||
| TestAllocatorManager.cc | ||
| TestAllocatorManager.h | ||