diff --git a/docs/build/custom.md b/docs/build/custom.md index 9a0c5b7770..c6aea5160d 100644 --- a/docs/build/custom.md +++ b/docs/build/custom.md @@ -169,16 +169,16 @@ pip install -U .\build\Windows\MinSizeRel\MinSizeRel\dist\onnxruntime-1.7.0-cp37 Unless there is a specific feature you need, do not use the unreleased `master` branch. -Once you have cloned the ONNX Runtime repo, checkout one of the release branches to build from. +Once you have cloned the ONNX Runtime repo, checkout one of the release tags to build from. ```bash git clone --recursive https://github.com/microsoft/onnxruntime cd onnxruntime -git checkout +git checkout ``` -Release branch names follow the pattern `rel-`. For example, `rel-1.11.0`. -Find them [here](https://github.com/microsoft/onnxruntime/branches/all?query=rel-). +Release tag names follow the pattern `v`. For example, `v1.11.0`. +Find them [here](https://github.com/microsoft/onnxruntime/tags). ## Example build commands diff --git a/docs/performance/float16.md b/docs/performance/float16.md index f36eef38f3..42d600a6cf 100644 --- a/docs/performance/float16.md +++ b/docs/performance/float16.md @@ -1,7 +1,7 @@ --- title: Create Float16 and Mixed Precision Models parent: How to -nav_order: 5 +nav_order: 4 --- # Create Float16 and Mixed Precision Models {: .no_toc } diff --git a/docs/performance/graph-optimizations.md b/docs/performance/graph-optimizations.md index 311433b7d8..cd1896bdd5 100644 --- a/docs/performance/graph-optimizations.md +++ b/docs/performance/graph-optimizations.md @@ -1,7 +1,7 @@ --- title: Graph optimizations parent: Performance -nav_order: 3 +nav_order: 2 redirect_from: /docs/how-to/graph-optimizations --- diff --git a/docs/performance/mobile-performance-tuning.md b/docs/performance/mobile-performance-tuning.md index 236396fc8d..c98066ed73 100644 --- a/docs/performance/mobile-performance-tuning.md +++ b/docs/performance/mobile-performance-tuning.md @@ -1,11 +1,12 @@ --- title: Tune Mobile Performance parent: Performance -nav_order: 2 -redirect_from: /docs/how-to/mobile-performance-tuning +nav_exclude: true --- {::options toc_levels="2" /} +# **IMPORTANT: THIS INFORMATION ONLY APPLIES TO ONNX RUNTIME VERSION 1.10 AND EARLIER. PLEASE USE A NEWER VERSION.** + # ONNX Runtime Mobile Performance Tuning Learn how different optimizations affect performance, and get suggestions for performance testing with ORT format models. diff --git a/docs/performance/quantization.md b/docs/performance/quantization.md index 09df778e15..70910d1e6c 100644 --- a/docs/performance/quantization.md +++ b/docs/performance/quantization.md @@ -1,7 +1,7 @@ --- title: Quantize ONNX Models parent: Performance -nav_order: 4 +nav_order: 3 redirect_from: /docs/how-to/quantization --- # Quantize ONNX Models diff --git a/docs/reference/mobile/helpers.md b/docs/reference/mobile/helpers.md index b7e07a0764..3d9c514d41 100644 --- a/docs/reference/mobile/helpers.md +++ b/docs/reference/mobile/helpers.md @@ -3,6 +3,7 @@ title: Model Export Helpers descriptions: Helpers to assist with export and usage of models with ORT Mobile parent: Mobile grand_parent: Reference +nav_order: 1 --- # ORT Mobile Model Export Helpers diff --git a/docs/reference/mobile/ort-format-model-runtime-optimization.md b/docs/reference/mobile/ort-format-model-runtime-optimization.md new file mode 100644 index 0000000000..f78df3f893 --- /dev/null +++ b/docs/reference/mobile/ort-format-model-runtime-optimization.md @@ -0,0 +1,59 @@ +--- +title: ORT Format Model Runtime Optimization +parent: Mobile +grand_parent: Reference +nav_order: 2 +--- +{::options toc_levels="2" /} + +# ORT Format Model Runtime Optimization + +## Contents +{: .no_toc } + +* TOC placeholder +{:toc} + +## Background + +The full ONNX Runtime build supports [graph optimizations](../../performance/graph-optimizations.md) at runtime for ONNX models. + +The ORT format model was designed to be used with ONNX Runtime [minimal builds](../../build/custom.md#minimal-build) for environments where smaller binary size is important. To reduce the binary size, some or all of the graph optimizer code is excluded from a minimal build. As such, ONNX models and ORT format models do not share the same graph optimization process. + +In ONNX Runtime **1.11 and later**, there is limited support for graph optimizations at runtime for ORT format models. This only applies to extended minimal builds or full builds. + +In ONNX Runtime **1.10 and earlier**, there is **no support** for graph optimizations at runtime for ORT format models. Any graph optimizations must be done at model conversion time. + +As a rule, [basic graph optimizations](../../performance/graph-optimizations.md#basic-graph-optimizations) are semantics-preserving and result in a valid ONNX graph. The basic optimizations can and generally should be baked in to the converted ORT format model at conversion time - this is the default behavior of the conversion script. In fact, any runtime optimization support for ORT format models will not include basic optimizations at all. + +## Types of runtime optimization + +These only apply to extended minimal builds or full builds. + +### Saved runtime optimizations + +Some graph optimizers support additional modes to save and load information about potential graph optimizations to and from the ORT format model. These potential optimizations are known as saved runtime optimizations. + +Saved runtime optimizations are only applied at runtime if they are still applicable. For example, a CPU Execution Provider (EP)-specific optimization for some nodes is only applicable if those nodes are assigned to the CPU EP at runtime. + +When converting from ONNX to ORT format, the potential optimizations are identified (1) and their effects are saved alongside the graph (without those optimizations applied) in the ORT format model. Later, when loading the ORT format model with saved runtime optimizations, the effects of potential optimizations are applied (2) if the potential optimizations are still applicable. + +In an extended minimal build, only enough implementation to support (2) is included, reducing the binary size. + +### Graph optimizers + +Some graph optimizers are also fully enabled in an extended minimal build and can be directly applied to an ORT format model. One example is the NHWC transformer. + +## Choosing whether to use runtime optimizations + +The use of runtime optimizations is optional. Generally, the choice is between using an ORT format model with saved runtime optimizations and using a fully optimized ORT format model. + +A fully optimized model will have the full set of ONNX Runtime optimizations (at the extended level or higher) available but will be fully optimized for the configuration at model conversion time. + +A model with saved runtime optimizations has fewer optimizations available but has more flexibility at runtime. For example, at runtime, a compiling EP like the NNAPI EP can claim the set of nodes it can handle and the remaining nodes can be further optimized with saved runtime optimizations. + +You can compare the performance of: +- A fully optimized model run with only the CPU EP enabled +- A model with saved runtime optimizations run with additional EPs enabled + +The [model usability checker](./model-usability-checker.md) will provide guidance for a particular model. diff --git a/docs/reference/ort-format-models.md b/docs/reference/ort-format-models.md index 034ae4933c..b4b595b4a4 100644 --- a/docs/reference/ort-format-models.md +++ b/docs/reference/ort-format-models.md @@ -22,26 +22,6 @@ The ORT format is the format supported by reduced size ONNX Runtime builds. Redu Both ORT format models and ONNX models are supported by a full ONNX Runtime build. -### Saved runtime optimizations - -**Since ONNX Runtime 1.11** - -An ORT format model may contain extra information about potential graph optimizations that may be applied at runtime. These are known as saved runtime optimizations. - -Saved runtime optimizations are only applied at runtime if they are still applicable. -For example, a CPU Execution Provider (EP)-specific optimization for some nodes is only be applicable if those nodes are assigned to the CPU EP at runtime. - -In a reduced size ONNX Runtime build, many of the graph optimizers are not supported the way they are in a full build because their complete implementations have significant binary size. -Consequently, the graph optimizers do not run the same way at runtime for ORT format models as they do for ONNX format models. - -To allow for dynamic graph optimization at runtime, some graph optimizers support additional modes to save and load saved runtime optimizations to and from the ORT format model. -When converting from ONNX to ORT format, the potential optimizations are identified (1) and their effects are saved alongside the graph (without those optimizations applied) in the ORT format model. -Later, when loading the ORT format model with saved runtime optimizations, the effects of potential optimizations are applied (2) if the potential optimizations are still applicable. - -In a reduced size build (an [extended minimal build](../build/custom.md#minimal-build)), only enough implementation to support (2) is included, reducing the graph optimizers' binary size. - -Note that saving runtime optimizations is optional. Alternatively, it is possible to save a fully optimized ORT format model, but the graph's optimizations may be hardware-dependent and less flexible at runtime. - ## Convert ONNX models to ORT format ONNX models are converted to ORT format using the `convert_onnx_models_to_ort` script. @@ -158,12 +138,7 @@ optional arguments: **Since ONNX Runtime 1.11** -Style of optimization to perform on the ORT format model. Multiple values may be provided. The conversion will run once for each value. The general guidance is to use models optimized with 'Runtime' style when using NNAPI or CoreML and 'Fixed' style otherwise. -- 'Fixed': Run optimizations directly before saving the ORT format model. This bakes in any platform-specific optimizations. -- 'Runtime': Run basic optimizations directly and save certain other optimizations to be applied at runtime if possible. This is useful when using a compiling EP like NNAPI or CoreML that may run an unknown (at model conversion time) number of nodes. The saved optimizations can further optimize nodes not assigned to the compiling EP at runtime. - See the documentation on [saved runtime optimizations](#saved-runtime-optimizations). - -See the documentation on [performance tuning mobile scenarios](../performance/mobile-performance-tuning.md) for more information. +Specify whether the converted model will be fully optimized ("Fixed") or have saved runtime optimizations ("Runtime"). Both types of models are produced by default. See [here](./mobile/ort-format-model-runtime-optimization.md) for more information. This replaces the [optimization level](#optimization-level) option from earlier ONNX Runtime versions.