saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-18 18:52:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add doco on session.use_ort_model_bytes_for_initializers (#12805)

Browse source
This commit is contained in:
Scott McKay 2022-09-01 06:17:48 +10:00 committed by GitHub
parent d83489c6f0
commit ddaecc489c
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 5 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
docs/reference/ort-format-models.md
View file

@ -239,7 +239,11 @@ const session = await ort.InferenceSession.create("<path to model>");
If a session is created using an input byte array containing the ORT format model data, by default we will copy the model bytes at the time of session creation to ensure the model bytes buffer is valid.
You may also enable the option to use the model bytes directly by setting the Session Options config `session.use_ort_model_bytes_directly` to `1`. This may reduce the peak memory usage of ONNX Runtime Mobile, but you will need to guarantee that the model bytes are valid throughout the lifespan of the ORT session. For ONNX Runtime Web, this option is set by default.
You may also enable the option to use the model bytes directly by setting the SessionOptions config entry `session.use_ort_model_bytes_directly` to `1`. This may reduce the peak memory usage of ONNX Runtime Mobile, but you will need to guarantee that the model bytes are valid throughout the lifespan of the ORT session. For ONNX Runtime Web, this option is set by default.
If `session.use_ort_model_bytes_directly` is enabled there is also an option to directly use the model bytes for initializers to further reduce peak memory usage. Set the Session Options config entry `session.use_ort_model_bytes_for_initializers` to `1` to enable this.
Note that if an initializer gets pre-packed it will undo the peak memory usage saving from using the model bytes directly for that initializer, as a new buffer for the pre-packed data needs to be allocated. Pre-packing is an optional performance optimization that involves changing the initializer layout to the optimal ordering for the current platform if it differs. If reducing peak memory usage is more important than potential performance optimizations pre-packing can be disabled by setting `session.disable_prepacking` to `1`.
C++ API
```c++