3681048474
### Description Staged: https://faxu.github.io/onnxruntime/docs/performance/ Main changes: - Restructure performance section to break into sub-categories - Move CUDA specific perf tuning tips to [CUDA EP page](https://faxu.github.io/onnxruntime/docs/execution-providers/CUDA-ExecutionProvider.html#performance-tuning) - Update [Transformer optimizer page](https://faxu.github.io/onnxruntime/docs/performance/transformers-optimization.html) to remove version-specific content... will be supported along with https://github.com/microsoft/onnxruntime/pull/14964 - Fix links to point to new pages
1.3 KiB
| title | grand_parent | parent | nav_order |
|---|---|---|---|
| Memory consumption | Performance | Tune performance | 2 |
Reduce memory consumption
Contents
{: .no_toc }
- TOC placeholder {:toc}
Shared arena based allocator
Memory consumption can be reduced between multiple sessions by configuring the shared arena based allocation. See the Share allocator(s) between sessions section in the C API documentation.
mimalloc allocator usage
ONNX Runtime supports overriding memory allocations using mimalloc, a fast, general-purpose allocator.
Depending on your model and usage, it can deliver single- or double-digit improvements in performance. The GitHub README page describes various scenarios on how mimalloc can be leveraged for performance tuning.
mimalloc is a submodule in the ONNX Runtime source tree. On Windows, one can employ the --use_mimalloc build flag which builds a static version of mimalloc and links it to ONNX Runtime. This redirects ONNX Runtime allocators and all new/delete calls to mimalloc.
Currently, there are no special provisions to employ mimalloc on Linux. It is recommended to use the LD_PRELOAD mechanism using pre-built binaries of mimalloc that you can build/obtain separately.