saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-18 18:52:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Fixed markdown syntax error and modify field name to keep the style consistent (#8659)

Browse source
This commit is contained in:
XinYuan 2022-06-06 11:34:31 +08:00 committed by GitHub
parent 9617040c63
commit 960f2a1a90
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
docs/get-started/with-c.md
View file

@ -87,8 +87,8 @@ chooses to override this by setting ```session_state.use_env_allocators``` to "0
* **Share initializer(s) and their ORT pre-processed version(s) between sessions:**
* *Description*: This feature allows a user to share the same instance of an initializer (and their ORT "pre-processed" versions)across multiple sessions.
* *Scenario*: You've several models that use the same set of initializers except the last few layers of the model and you load these models in the same process. When every model (session) creates a separate instance of the same initializer, it leads to excessive and wasteful memory usage since in this case it's the same initializer. You want to optimize memory usage while having the flexibility to allocate the initializers (possibly even store them in shared memory).
* *Example Usage*: Use the ```AddInitializer``` API to add a pre-allocated initializer to session options before calling ```CreateSession```. Use the same instance of session options to create several sessions allowing the initializer(s) to be shared between the sessions. See [C API sample usage (TestSharingOfInitializerAndItsPrepackedVersion)](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/test/shared_lib/test_inference.cc) and [C# API sample usage (TestSharingOfInitializerAndItsPrepackedVersion)](https://github.com/microsoft/onnxruntime/blob/master/csharp/test/Microsoft.ML.OnnxRuntime.Tests/InferenceTest.cs).
* *In some ORT operator implementations, initializers are pre-processed when the model is loaded (a process called "pre-packing") to promote optimal operator inferencing on some platforms. By default, these pre-processed versions of initializers are maintained on a per-session basis (i.e.) they are not shared between sessions. To enable sharing these between sessions, create a container (using ```CreatePrepackedWeightsContainer```) and pass this at session creation time so that the sharing of pre-packed versions of shared initializers between sessions take place and these are not duplicated in memory. The same tests referenced above in C and C# shows sample usage of this feature as well.
* *Usage*: Use the ```AddInitializer``` API to add a pre-allocated initializer to session options before calling ```CreateSession```. Use the same instance of session options to create several sessions allowing the initializer(s) to be shared between the sessions. See [C API sample usage (TestSharingOfInitializerAndItsPrepackedVersion)](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/test/shared_lib/test_inference.cc) and [C# API sample usage (TestSharingOfInitializerAndItsPrepackedVersion)](https://github.com/microsoft/onnxruntime/blob/master/csharp/test/Microsoft.ML.OnnxRuntime.Tests/InferenceTest.cs).
* In some ORT operator implementations, initializers are pre-processed when the model is loaded (a process called "pre-packing") to promote optimal operator inferencing on some platforms. By default, these pre-processed versions of initializers are maintained on a per-session basis (i.e.) they are not shared between sessions. To enable sharing these between sessions, create a container (using ```CreatePrepackedWeightsContainer```) and pass this at session creation time so that the sharing of pre-packed versions of shared initializers between sessions take place and these are not duplicated in memory. The same tests referenced above in C and C# shows sample usage of this feature as well.
NOTE: Any kernel developer wishing to implement pre-packing MUST write a test that triggers pre-packing of all weights that can be possibly pre-packed using the kernel and must test sharing of these pre-packed weights between sessions. See [kernel test (SharedPrepackedWeights)](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/test/providers/cpu/math/gemm_test.cc).