From 645ebcb5af34e5318ad49a65d864d13cd5ef8554 Mon Sep 17 00:00:00 2001 From: Edward Chen <18449977+edgchen1@users.noreply.github.com> Date: Wed, 6 Apr 2022 09:50:44 -0700 Subject: [PATCH] ORT 1.11 mobile doc updates (#11062) * Save updates. * save changes * Update install matrix text. * save updates * Rename ort-model-format.md to ort-format-models.md. * Remove redirect_from entry, doesn't seem to work. * PR comments. * Reword. --- docs/build/custom.md | 44 +++++----- docs/build/web.md | 2 +- .../CoreML-ExecutionProvider.md | 5 +- .../NNAPI-ExecutionProvider.md | 7 +- docs/install/index.md | 24 ++++-- docs/performance/mobile-performance-tuning.md | 4 +- docs/reference/build-web-app.md | 2 +- ...t-model-format.md => ort-format-models.md} | 84 +++++++++++++------ .../reference/reduced-operator-config-file.md | 2 +- docs/tutorials/mobile/deploy-ios.md | 4 +- docs/tutorials/mobile/index.md | 48 +++++++++-- .../classify-images-nextjs-github-template.md | 2 +- docs/tutorials/web/index.md | 2 +- js/script.js | 28 +++---- 14 files changed, 168 insertions(+), 90 deletions(-) rename docs/reference/{ort-model-format.md => ort-format-models.md} (62%) diff --git a/docs/build/custom.md b/docs/build/custom.md index 1734001c37..9a0c5b7770 100644 --- a/docs/build/custom.md +++ b/docs/build/custom.md @@ -47,7 +47,7 @@ The operators that are included are specified at build time, in a [configuration **`--enable_reduced_operator_type_support`** -* Enables [operator type reduction](../reference/ort-model-format.md#enable-type-reduction). Requires ONNX Runtime version 1.7 or higher and for type reduction to have been enabled during model conversion +* Enables [operator type reduction](../reference/ort-format-models.md#enable-type-reduction). Requires ONNX Runtime version 1.7 or higher and for type reduction to have been enabled during model conversion If the configuration file is created using ORT format models, the input/output types that individual operators require can be tracked if `--enable_type_reduction` is specified. This can be used to further reduce the build size if `--enable_reduced_operator_type_support` is specified when building ORT. @@ -55,21 +55,30 @@ ONNX format models are not guaranteed to include the required per-node type info ## Minimal build -ONNX Runtime can be built to further minimize the binary size, by only including support for loading and executing models in [ORT format](../reference/ort-model-format.md), and not ONNX format. +ONNX Runtime can be built to further minimize the binary size. +These reduced size builds are called minimal builds and there are different minimal build levels described below. + +### Basic **`--minimal_build`** -* RTTI is disabled by default in this build, unless the Python bindings (`--build_wheel`) are enabled. -* If you wish to enable execution providers that compile kernels such as NNAPI or CoreML specify `--minimal_build extended`. See [here](../execution-providers/NNAPI-ExecutionProvider.md) and [here](../execution-providers/CoreML-ExecutionProvider.md) for details on using NNAPI and CoreML with ONNX Runtime Mobile +RTTI is disabled by default in this build, unless the Python bindings (`--build_wheel`) are enabled. -A minimal build has the following limitations: +A basic minimal build has the following limitations: -* No support for ONNX format models, that is model must be converted to ORT format -* No support for runtime optimizations. Optimizations are performed during conversion to ORT format -* Limited support for runtime partitioning (assigning nodes in a model to specific execution providers). Execution providers that statically register kernels (e.g. ONNX Runtime CPU Execution Provider) are supported by default. All execution providers that will be used at runtime MUST be registered when creating the ORT format model - - Execution providers that compile nodes are optionally supported - - currently this is limited to the NNAPI and CoreML Execution Providers +* No support for ONNX format models. The model must be converted to [ORT format](../reference/ort-format-models.md). +* No support for runtime optimizations. Optimizations are performed during conversion to ORT format. +* Support for execution providers that statically register kernels (e.g. ONNX Runtime CPU Execution Provider) only. +### Extended + +**`--minimal_build extended`** + +An extended minimal build supports more functionality than a basic minimal build: + +* Limited support for runtime partitioning (assigning nodes in a model to specific execution providers). +* Additional support for execution providers that compile kernels such as [NNAPI](../execution-providers/NNAPI-ExecutionProvider.md) and [CoreML](../execution-providers/CoreML-ExecutionProvider.md). +* **ONNX Runtime version 1.11 and later**: Limited support for runtime optimizations, via saved runtime optimizations and a few graph optimizers that are enabled at runtime. ## Other customizations @@ -112,7 +121,7 @@ _[This section is coming soon]_ ### iOS -To produce pods for an iOS build, the build options are wrapped in the [build_and_assemble_ios_pods.py](https://github.com/microsoft/onnxruntime/blob/master/tools/ci_build/github/apple/build_ios_framework.py) script. +To produce pods for an iOS build, use the [build_and_assemble_ios_pods.py](https://github.com/microsoft/onnxruntime/blob/master/tools/ci_build/github/apple/build_and_assemble_ios_pods.py) script. Example usage: @@ -127,7 +136,7 @@ This will do a custom build and create the pod package files for it in /path/to/ Next, update the Podfile to use the local pods: -``` +```diff - pod 'onnxruntime-mobile-objc' + pod 'onnxruntime-mobile-objc', :path => "/path/to/staging/dir/onnxruntime-mobile-objc" + pod 'onnxruntime-mobile-c', :path => "/path/to/staging/dir/onnxruntime-mobile-c" @@ -168,15 +177,8 @@ cd onnxruntime git checkout ``` -where `release branch` is one of the values in the `Branch` column: - -| Release | Date | Branch | -|---------|------|--------| -| 1.9 | 2021-09-22 | rel-1.9.1 | -| 1.8 | 2021-06-02 | rel-1.8.2 | -| 1.7 | 2021-03-03 | rel-1.7.2 | -| 1.6 | 2020-12-11 | rel-1.6.0 | -| 1.5 | 2020-10-30 | rel-1.5.3 | +Release branch names follow the pattern `rel-`. For example, `rel-1.11.0`. +Find them [here](https://github.com/microsoft/onnxruntime/branches/all?query=rel-). ## Example build commands diff --git a/docs/build/web.md b/docs/build/web.md index 4aaa1acdfb..b3905043c5 100644 --- a/docs/build/web.md +++ b/docs/build/web.md @@ -77,7 +77,7 @@ To get all build artifacts of ONNX Runtime WebAssembly, it needs 4 times of buil ### Minimal Build Support -ONNX Runtime WebAssembly can be built with flag `--minimal_build`. This will generate smaller artifacts and also have a less runtime memory usage. An ORT format model is required. A detailed instruction will come soon. See also [ORT format Conversion](../reference/ort-model-format.md). +ONNX Runtime WebAssembly can be built with flag `--minimal_build`. This will generate smaller artifacts and also have a less runtime memory usage. An ORT format model is required. A detailed instruction will come soon. See also [ORT format Conversion](../reference/ort-format-models.md). ### FAQ diff --git a/docs/execution-providers/CoreML-ExecutionProvider.md b/docs/execution-providers/CoreML-ExecutionProvider.md index a4d1ad2a56..3b8157263e 100644 --- a/docs/execution-providers/CoreML-ExecutionProvider.md +++ b/docs/execution-providers/CoreML-ExecutionProvider.md @@ -25,10 +25,9 @@ It is recommended to use Apple devices equipped with Apple Neural Engine to achi ## Install -Pre-built binaries of ONNX Runtime Mobile with CoreML EP for iOS are published to CocoaPods. +Pre-built binaries of ONNX Runtime with CoreML EP for iOS are published to CocoaPods. -- C/C++: **onnxruntime-mobile-c** -- Objective-C: **onnxruntime-mobile-objc** +See [here](../install/index.md#install-on-ios) for installation instructions. ## Build diff --git a/docs/execution-providers/NNAPI-ExecutionProvider.md b/docs/execution-providers/NNAPI-ExecutionProvider.md index f838ddf3bc..16227517ae 100644 --- a/docs/execution-providers/NNAPI-ExecutionProvider.md +++ b/docs/execution-providers/NNAPI-ExecutionProvider.md @@ -22,7 +22,9 @@ Accelerate ONNX models on Android devices with ONNX Runtime and the NNAPI execut The NNAPI Execution Provider (EP) requires Android devices with Android 8.1 or higher. It is recommended to use Android devices with Android 9 or higher to achieve optimal performance. ## Install -Pre-built packages of ONNX Runtime Mobile with NNAPI EP for Android are published on Maven: [com.microsoft.onnxruntime:onnxruntime-mobile](https://mvnrepository.com/artifact/com.microsoft.onnxruntime/onnxruntime-mobile). +Pre-built packages of ONNX Runtime with NNAPI EP for Android are published on Maven. + +See [here](../install/index.md#install-on-android) for installation instructions. ## Build @@ -136,6 +138,3 @@ Following ops are supported by the NNAPI Execution Provider, |com.microsoft:QLinearAdd|All quantization scales and zero points should be constant.| |com.microsoft:QLinearAveragePool|Only 2D Pool is supported.
All quantization scales and zero points should be constant.| |com.microsoft:QLinearSigmoid|All quantization scales and zero points should be constant.| - - - diff --git a/docs/install/index.md b/docs/install/index.md index e3ddedfb91..7a4b1612c2 100644 --- a/docs/install/index.md +++ b/docs/install/index.md @@ -72,9 +72,13 @@ dotnet add package Microsoft.AI.MachineLearning ## Install on web and mobile -The installation instructions in this section use pre-built packages that include support for selected operators and ONNX opset versions based on the requirements of popular models. Your model must only use the [opsets and operators](../reference/operators/mobile_package_op_type_support_1.9.md) supported by the pre-built package. +Unless stated otherwise, the installation instructions in this section refer to pre-built packages that include support for selected operators and ONNX opset versions based on the requirements of popular models. These packages may be referred to as "mobile packages". If you use mobile packages, your model must only use the supported [opsets and operators](../reference/operators/mobile_package_op_type_support_1.9.md). -If the pre-built package is too large, or does not include the operators in your model/s, you can create a [custom build](../build/custom.md). +Another type of pre-built package has full support for all ONNX opsets and operators, at the cost of larger binary size. These packages are referred to as "full packages". + +If the pre-built mobile package supports your model/s but is too large, you can create a [custom build](../build/custom.md). A custom build can include just the opsets and operators in your model/s to reduce the size. + +If the pre-built mobile package does not include the opsets or operators in your model/s, you can either use the full package if available, or create a custom build. ### JavaScript Installs @@ -105,14 +109,16 @@ npm install onnxruntime-react-native ### Install on iOS -In your CocoaPods `Podfile`, add the `onnxruntime-mobile-c` or `onnxruntime-mobile-objc` pod depending on which API you wish to use. +In your CocoaPods `Podfile`, add the `onnxruntime-c`, `onnxruntime-mobile-c`, `onnxruntime-objc`, or `onnxruntime-mobile-objc` pod, depending on whether you want to use a full or mobile package and which API you want to use. #### C/C++ ```pod use_frameworks! - pod 'onnxruntime-mobile-c' + # choose one of the two below: + pod 'onnxruntime-c' # full package + #pod 'onnxruntime-mobile-c' # mobile package ``` #### Objective-C @@ -120,7 +126,9 @@ In your CocoaPods `Podfile`, add the `onnxruntime-mobile-c` or `onnxruntime-mobi ```pod use_frameworks! - pod 'onnxruntime-mobile-objc' + # choose one of the two below: + pod 'onnxruntime-objc' # full package + #pod 'onnxruntime-mobile-objc' # mobile package ``` Run `pod install`. @@ -143,13 +151,15 @@ In your Android Studio Project, make the following changes to: ```gradle dependencies { - implementation 'com.microsoft.onnxruntime:onnxruntime-mobile:' + // choose one of the two below: + implementation 'com.microsoft.onnxruntime:onnxruntime-android:latest.release' // full package + //implementation 'com.microsoft.onnxruntime:onnxruntime-mobile:latest.release' // mobile package } ``` #### C/C++ -Download the onnxruntime-mobile AAR hosted at [MavenCentral](https://mvnrepository.com/artifact/com.microsoft.onnxruntime/onnxruntime-mobile), change the file extension from `.aar` to `.zip`, and unzip it. Include the header files from the `headers` folder, and the relevant `libonnxruntime.so` dynamic library from the `jni` folder in your NDK project. +Download the [onnxruntime-android](https://mvnrepository.com/artifact/com.microsoft.onnxruntime/onnxruntime-android) (full package) or [onnxruntime-mobile](https://mvnrepository.com/artifact/com.microsoft.onnxruntime/onnxruntime-mobile) (mobile package) AAR hosted at MavenCentral, change the file extension from `.aar` to `.zip`, and unzip it. Include the header files from the `headers` folder, and the relevant `libonnxruntime.so` dynamic library from the `jni` folder in your NDK project. ## ORT Training package diff --git a/docs/performance/mobile-performance-tuning.md b/docs/performance/mobile-performance-tuning.md index 081d27d378..236396fc8d 100644 --- a/docs/performance/mobile-performance-tuning.md +++ b/docs/performance/mobile-performance-tuning.md @@ -48,7 +48,7 @@ _Layout_ optimizations may be hardware specific and involve internal conversions ### Outcome of optimizations when creating an optimized ORT format model -Below is an example of the changes that occur in _basic_ and _extended_ optimizations when applied to the MNIST model with only the CPU EP enabled. The optimization level is specified when [creating the ORT format model](../reference/ort-model-format.md#optimization-level). +Below is an example of the changes that occur in _basic_ and _extended_ optimizations when applied to the MNIST model with only the CPU EP enabled. The optimization level is specified when [creating the ORT format model](../reference/ort-format-models.md#optimization-level). - At the _basic_ level we combine the Conv and Add nodes (the addition is done via the 'B' input to Conv), we combine the MatMul and Add into a single Gemm node (the addition is done via the 'C' input to Gemm), and constant fold to remove one of the Reshape nodes. - `python /tools/python/convert_onnx_models_to_ort.py --optimization_level basic /dir_with_mnist_onnx_model` @@ -121,7 +121,7 @@ To create an NNAPI-aware ORT format model please follow these steps. pip install -U build\Windows\RelWithDebIfo\RelWithDebIfo\dist\onnxruntime_noopenmp-1.7.0-cp37-cp37m-win_amd64.whl ``` -3. Create an NNAPI-aware ORT format model by running `convert_onnx_models_to_ort.py` as per the [standard instructions](../reference/ort-model-format.md), with NNAPI enabled (`--use_nnapi`), and the optimization level set to _extended_ or _all_ (e.g. `--optimization_level extended`). This will allow higher level optimizations to run on any nodes that NNAPI can not handle. +3. Create an NNAPI-aware ORT format model by running `convert_onnx_models_to_ort.py` as per the [standard instructions](../reference/ort-format-models.md), with NNAPI enabled (`--use_nnapi`), and the optimization level set to _extended_ or _all_ (e.g. `--optimization_level extended`). This will allow higher level optimizations to run on any nodes that NNAPI can not handle. ``` python /tools/python/convert_onnx_models_to_ort.py --use_nnapi --optimization_level extended /models ``` diff --git a/docs/reference/build-web-app.md b/docs/reference/build-web-app.md index 5e843036db..3a786ff735 100644 --- a/docs/reference/build-web-app.md +++ b/docs/reference/build-web-app.md @@ -47,7 +47,7 @@ You need to understand your web app's scenario and get an ONNX model that is app ONNX models can be obtained from the [ONNX model zoo](https://github.com/onnx/models), converted from PyTorch or TensorFlow, and many other places. -You can [convert the ONNX format model to ORT format model](./ort-model-format.md), for optimized binary size, faster initialization and peak memory usage. +You can [convert the ONNX format model to ORT format model](./ort-format-models.md), for optimized binary size, faster initialization and peak memory usage. You can [perform a model-specific custom build](../build/custom.md) to further optimize binary size. diff --git a/docs/reference/ort-model-format.md b/docs/reference/ort-format-models.md similarity index 62% rename from docs/reference/ort-model-format.md rename to docs/reference/ort-format-models.md index eb780368b2..034ae4933c 100644 --- a/docs/reference/ort-model-format.md +++ b/docs/reference/ort-format-models.md @@ -1,6 +1,6 @@ --- title: ORT format models -description: Define the ORT model format and show how to convert an ONNX model to ORT format to run on mobile or web +description: Define the ORT format and show how to convert an ONNX model to ORT format to run on mobile or web parent: Reference has_children: false nav_order: 4 @@ -16,9 +16,31 @@ redirect_from: /docs/tutorials/mobile/model-conversion, /docs/tutorials/mobile/m * TOC {:toc} -## What is the ORT model format? +## What is the ORT format? -The ORT model format is a reduced-size model format used in constrained environments such as mobile and web applications. ONNX Runtime provides tools to convert ONNX models to ORT format. +The ORT format is the format supported by reduced size ONNX Runtime builds. Reduced size builds may be more appropriate for use in size-constrained environments such as mobile and web applications. + +Both ORT format models and ONNX models are supported by a full ONNX Runtime build. + +### Saved runtime optimizations + +**Since ONNX Runtime 1.11** + +An ORT format model may contain extra information about potential graph optimizations that may be applied at runtime. These are known as saved runtime optimizations. + +Saved runtime optimizations are only applied at runtime if they are still applicable. +For example, a CPU Execution Provider (EP)-specific optimization for some nodes is only be applicable if those nodes are assigned to the CPU EP at runtime. + +In a reduced size ONNX Runtime build, many of the graph optimizers are not supported the way they are in a full build because their complete implementations have significant binary size. +Consequently, the graph optimizers do not run the same way at runtime for ORT format models as they do for ONNX format models. + +To allow for dynamic graph optimization at runtime, some graph optimizers support additional modes to save and load saved runtime optimizations to and from the ORT format model. +When converting from ONNX to ORT format, the potential optimizations are identified (1) and their effects are saved alongside the graph (without those optimizations applied) in the ORT format model. +Later, when loading the ORT format model with saved runtime optimizations, the effects of potential optimizations are applied (2) if the potential optimizations are still applicable. + +In a reduced size build (an [extended minimal build](../build/custom.md#minimal-build)), only enough implementation to support (2) is included, reducing the graph optimizers' binary size. + +Note that saving runtime optimizations is optional. Alternatively, it is possible to save a fully optimized ORT format model, but the graph's optimizations may be hardware-dependent and less flexible at runtime. ## Convert ONNX models to ORT format @@ -94,50 +116,64 @@ where: The current optional arguments are available by running the script with the `--help` argument. Supported arguments and defaults differ slightly across ONNX Runtime versions. +Help text from ONNX Runtime 1.11: ```bash python -m onnxruntime.tools.convert_onnx_models_to_ort --help ``` ```output -usage: convert_onnx_models_to_ort.py [-h] [--use_nnapi] - [--optimization_level {disable,basic,extended,all}] - [--enable_type_reduction] - [--custom_op_library CUSTOM_OP_LIBRARY] [--save_optimized_onnx_model] - model_path_or_dir +usage: convert_onnx_models_to_ort.py [-h] [--optimization_style {Fixed,Runtime} [{Fixed,Runtime} ...]] [--enable_type_reduction] [--custom_op_library CUSTOM_OP_LIBRARY] [--save_optimized_onnx_model] [--allow_conversion_failures] [--nnapi_partitioning_stop_ops NNAPI_PARTITIONING_STOP_OPS] + [--target_platform {arm,amd64}] + model_path_or_dir -Convert the ONNX model/s in the provided directory to ORT format models. -All files with a `.onnx` extension will be processed. For each one, an ORT format model will be created in the same directory. -A configuration file will also be created called `required_operators.config`, and will contain the list of required operators for all converted models. -This configuration file should be used as input to the minimal build via the `--include_ops_by_config` parameter. +Convert the ONNX format model/s in the provided directory to ORT format models. All files with a `.onnx` extension will be processed. For each one, an ORT format model will be created in the same directory. A configuration file will also be created containing the list of required +operators for all converted models. This configuration file should be used as input to the minimal build via the `--include_ops_by_config` parameter. positional arguments: - model_path_or_dir Provide path to ONNX model or directory containing ONNX model/s to convert. All files with a .onnx extension, including in subdirectories, will be processed. + model_path_or_dir Provide path to ONNX model or directory containing ONNX model/s to convert. All files with a .onnx extension, including those in subdirectories, will be processed. optional arguments: -h, --help show this help message and exit - --optimization_level {disable,basic,extended,all} - Level to optimize ONNX model with, prior to converting to ORT format model. - These map to the onnxruntime.GraphOptimizationLevel values. - If the level is 'all' the NCHWc transformer is manually disabled as it contains device specific logic, - so the ORT format model must be generated on the device it will run on. - Additionally, the NCHWc optimizations are not applicable to ARM devices. + --optimization_style {Fixed,Runtime} [{Fixed,Runtime} ...] + Style of optimization to perform on the ORT format model. Multiple values may be provided. The conversion will run once for each value. The general guidance is to use models optimized with 'Runtime' style when using NNAPI or CoreML and 'Fixed' style otherwise. + 'Fixed': Run optimizations directly before saving the ORT format model. This bakes in any platform-specific optimizations. 'Runtime': Run basic optimizations directly and save certain other optimizations to be applied at runtime if possible. This is useful when + using a compiling EP like NNAPI or CoreML that may run an unknown (at model conversion time) number of nodes. The saved optimizations can further optimize nodes not assigned to the compiling EP at runtime. --enable_type_reduction - Add operator specific type information to the configuration file to potentially - reduce the types supported by individual operator implementations. + Add operator specific type information to the configuration file to potentially reduce the types supported by individual operator implementations. --custom_op_library CUSTOM_OP_LIBRARY Provide path to shared library containing custom operator kernels to register. --save_optimized_onnx_model - Save the optimized version of each ONNX model. This will have the same optimizations - applied as the ORT format model. + Save the optimized version of each ONNX model. This will have the same level of optimizations applied as the ORT format model. + --allow_conversion_failures + Whether to proceed after encountering model conversion failures. + --nnapi_partitioning_stop_ops NNAPI_PARTITIONING_STOP_OPS + Specify the list of NNAPI EP partitioning stop ops. In particular, specify the value of the "ep.nnapi.partitioning_stop_ops" session options config entry. + --target_platform {arm,amd64} + Specify the target platform where the exported model will be used. This parameter can be used to choose between platform-specific options, such as QDQIsInt8Allowed(arm), NCHWc (amd64) and NHWC (arm/amd64) format, different optimizer level options, etc. ``` #### Optional script arguments +##### Optimization style + +**Since ONNX Runtime 1.11** + +Style of optimization to perform on the ORT format model. Multiple values may be provided. The conversion will run once for each value. The general guidance is to use models optimized with 'Runtime' style when using NNAPI or CoreML and 'Fixed' style otherwise. +- 'Fixed': Run optimizations directly before saving the ORT format model. This bakes in any platform-specific optimizations. +- 'Runtime': Run basic optimizations directly and save certain other optimizations to be applied at runtime if possible. This is useful when using a compiling EP like NNAPI or CoreML that may run an unknown (at model conversion time) number of nodes. The saved optimizations can further optimize nodes not assigned to the compiling EP at runtime. + See the documentation on [saved runtime optimizations](#saved-runtime-optimizations). + +See the documentation on [performance tuning mobile scenarios](../performance/mobile-performance-tuning.md) for more information. + +This replaces the [optimization level](#optimization-level) option from earlier ONNX Runtime versions. + ##### Optimization level +**ONNX Runtime version 1.10 and earlier** + Set the optimization level that ONNX Runtime will use to optimize the model prior to saving in ORT format. -For ONNX Runtime version 1.8 and later, *all* is recommended if the model will be run with the CPU Execution Provider (EP). +For ONNX Runtime version 1.8 and later, *all* is recommended if the model will be run with the CPU EP. For earlier versions, *extended* is recommended, as the *all* level previously included device specific optimizations that would limit the portability of the model. diff --git a/docs/reference/reduced-operator-config-file.md b/docs/reference/reduced-operator-config-file.md index 05807112c8..6d32e0c723 100644 --- a/docs/reference/reduced-operator-config-file.md +++ b/docs/reference/reduced-operator-config-file.md @@ -11,7 +11,7 @@ nav_order: 3 The reduced operator config file is an input to the ONNX Runtime build-from-source script. It specifies which operators are included in the runtime. A reduced set of operators in ONNX Runtime permits a smaller build binary size. A smaller runtime is used in constrained environments, such as mobile and web deployments. -This article shows you how to generate the reduced operator config file using the `create_reduced_build_config.py` script. You can also generate the reduced operator config file by [converting ONNX models to ORT format](./ort-model-format.md). +This article shows you how to generate the reduced operator config file using the `create_reduced_build_config.py` script. You can also generate the reduced operator config file by [converting ONNX models to ORT format](./ort-format-models.md). ## Contents {: .no_toc} diff --git a/docs/tutorials/mobile/deploy-ios.md b/docs/tutorials/mobile/deploy-ios.md index 724d5fe9f1..8fa4f2a2d7 100644 --- a/docs/tutorials/mobile/deploy-ios.md +++ b/docs/tutorials/mobile/deploy-ios.md @@ -69,7 +69,7 @@ Here's an example screenshot of the app: > Conversion of this model is a two part process. The original model is in tflite format. This is firstly converted to ONNX format using the [tf2onnx converter](https://github.com/onnx/tensorflow-onnx). > - > The model is then converted into ORT format using the [onnx to ort converter](../../reference/ort-model-format.md). + > The model is then converted into ORT format using the [onnx to ort converter](../../reference/ort-format-models.md). > > As well as generating the model in ORT format, the conversion script also outputs an [operator config file](../../reference/reduced-operator-config-file.md) @@ -118,5 +118,5 @@ Here's an example screenshot of the app: * For more information on converting tflite models to ONNX [Converting TFlite models to ONNX](https://github.com/onnx/tensorflow-onnx#getting-started) -* [Convert ONNX models to ORT format](../../reference/ort-model-format.md) +* [Convert ONNX models to ORT format](../../reference/ort-format-models.md) diff --git a/docs/tutorials/mobile/index.md b/docs/tutorials/mobile/index.md index 348d9ff1ac..4370af4528 100644 --- a/docs/tutorials/mobile/index.md +++ b/docs/tutorials/mobile/index.md @@ -18,13 +18,41 @@ ONNX Runtime gives you a variety of options to add machine learning to your mobi ![Steps to build for mobile platforms](../../../images/mobile.png){:width="60%"} -1. Which ONNX Runtime mobile library should I use? +1. Which ONNX Runtime package should I use? - We publish the following ONNX Runtime mobile libraries: - * Android C/C++ - * Android Java + We publish the following ONNX Runtime packages that can be used in mobile applications: + * Android Java/C/C++ + * Mobile (onnxruntime-mobile) and full (onnxruntime-android) packages * iOS C/C++ - * iOS Objective C + * Mobile (onnxruntime-mobile-c) and full (onnxruntime-c) packages + * iOS Objective-C + * Mobile (onnxruntime-mobile-objc) and full (onnxruntime-objc) packages + + The full package has the full ONNX Runtime feature set. + The mobile package has a smaller binary size but limited feature support, like a reduced set of operator implementations and no support for running ONNX models. + + A [custom build](../../build/custom.md) is tailored to your model(s) and can be even smaller than the mobile package. However, using a custom build is more involved than using one of the published packages. + + If the binary size of the full package is acceptable, using the full package is recommended because it is easier to use. + Otherwise, consider using the mobile package or a custom build. + + To give an idea of the binary size difference between mobile and full packages: + + ONNX Runtime 1.11.0 Android package `jni/arm64-v8a/libonnxruntime.so` dynamic library file size: + |Package|Size| + |-|-| + |onnxruntime-mobile|3.3 MB| + |onnxruntime-android|12 MB| + + ONNX Runtime 1.11.0 iOS package `onnxruntime.xcframework/ios-arm64/onnxruntime.framework/onnxruntime` static library file size: + |Package|Size| + |-|-| + |onnxruntime-mobile-c|22 MB| + |onnxruntime-c|48 MB| + + Note: The iOS package is a static framework that will have a reduced binary size impact when compiled into your app. + + See [here](../../install/index.md#install-on-web-and-mobile) for installation instructions. 2. Which machine learning model does my application use? @@ -32,7 +60,7 @@ ONNX Runtime gives you a variety of options to add machine learning to your mobi ONNX models can be obtained from the [ONNX model zoo](https://github.com/onnx/models), converted from PyTorch or TensorFlow, and many other places. - Once you have sourced or converted the model into ONNX format, there is a further step required to optimize the model for mobile deployments. [Convert the model to ORT format](../../reference/ort-model-format.md) for optimized model binary size, faster initialization and peak memory usage. + Once you have sourced or converted the model into ONNX format, it must be [converted to an ORT format model](../../reference/ort-format-models.md#convert-onnx-models-to-ort-format) in order to be used with the ONNX Runtime mobile package. This conversion is not necessary if you are using the full package. 3. How do I bootstrap my app development? @@ -44,6 +72,10 @@ ONNX Runtime gives you a variety of options to add machine learning to your mobi 4. How do I optimize my application? - The execution environment on mobile devices has fixed memory and disk storage. Therefore, it is essential that any AI execution library is optimized to consume minimum resources in terms of disk footprint, memory and network usage (both model size and binary size). + To reduce binary size: - ONNX Runtime Mobile uses the ORT model format which enables us to create a [custom ORT build](../../build/custom.md) that minimizes the binary size and reduces memory usage for client side inference. The ORT model format file is generated from the regular ONNX model using the `onnxruntime` python package. The custom build does this primarily by only including specified operators and types in the build, as well as trimming down dependencies per custom needs. + Use the ONNX Runtime mobile package or a custom build to reduce the binary size. The mobile package requires use of an ORT format model. + + To reduce memory usage: + + Use an ORT format model as that uses less memory. diff --git a/docs/tutorials/web/classify-images-nextjs-github-template.md b/docs/tutorials/web/classify-images-nextjs-github-template.md index 5394665454..a558cb87b5 100644 --- a/docs/tutorials/web/classify-images-nextjs-github-template.md +++ b/docs/tutorials/web/classify-images-nextjs-github-template.md @@ -35,7 +35,7 @@ This application performs inference on device, in the browser using the onnxrunt We will be using [SqueezeNet](https://github.com/onnx/models/tree/master/vision/classification/squeezenet) from the [ONNX Model Zoo](https://github.com/onnx/models). SqueezeNet models perform image classification - they take images as input and classify the major object in the image into a set of pre-defined classes. They are trained on the ImageNet dataset which contains images from 1000 different classes. SqueezeNet models are highly efficient in terms of size and speed while providing good accuracies. This makes them ideal for platforms with strict constraints on size, like client side inference. - > If you need even more model memory and disk efficiency, you can convert the ONNX model to [ORT format](../../reference/ort-model-format) and use an ORT model in your application instead of the ONNX one. You can also also [reduce the size of the ONNX Runtime](../../build/custom.md) binary itself to only include support for the specific models in your application. + > If you need even more model memory and disk efficiency, you can convert the ONNX model to [ORT format](../../reference/ort-format-models) and use an ORT model in your application instead of the ONNX one. You can also also [reduce the size of the ONNX Runtime](../../build/custom.md) binary itself to only include support for the specific models in your application. ## Create a Static Site with NextJS (a ReactJS Framework) to deploy models in the browser diff --git a/docs/tutorials/web/index.md b/docs/tutorials/web/index.md index 576b7e2432..3d53ddb53f 100644 --- a/docs/tutorials/web/index.md +++ b/docs/tutorials/web/index.md @@ -76,7 +76,7 @@ For more detail on the steps below, see the [build a web application](../../refe The libraries and models mentioned in the previous steps can be optimized to meet memory and processing demands. - a. Models in ONNX format can be [converted to ORT format](../../reference/ort-model-format.md), for optimized model binary size, faster initialization and peak memory usage. + a. Models in ONNX format can be [converted to ORT format](../../reference/ort-format-models.md), for optimized model binary size, faster initialization and peak memory usage. b. The size of the ONNX Runtime itself can reduced by [building a custom package](../../build/custom.md) that only includes support for your specific model/s. diff --git a/js/script.js b/js/script.js index 81eb8ab4b6..65efb96e81 100644 --- a/js/script.js +++ b/js/script.js @@ -969,7 +969,7 @@ var validCombos = { "Follow build instructions from here", "android,Java,ARM64,NNAPI": - "Add a dependency on com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", + "Add a dependency on com.microsoft.onnxruntime:onnxruntime-android or com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", "android,C-API,X86,NNAPI": "Follow build instructions from here", @@ -978,7 +978,7 @@ var validCombos = { "Follow build instructions from here", "android,Java,X64,NNAPI": - "Add a dependency on com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", + "Add a dependency on com.microsoft.onnxruntime:onnxruntime-android or com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", "android,C-API,X64,NNAPI": "Follow build instructions from here", @@ -987,7 +987,7 @@ var validCombos = { "Follow build instructions from here", "android,Java,X86,NNAPI": - "Add a dependency on com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", + "Add a dependency on com.microsoft.onnxruntime:onnxruntime-android or com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", "android,C-API,ARM32,NNAPI": "Follow build instructions from here", @@ -996,7 +996,7 @@ var validCombos = { "Follow build instructions from here", "android,Java,ARM32,NNAPI": - "Add a dependency on com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", + "Add a dependency on com.microsoft.onnxruntime:onnxruntime-android or com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", "android,C-API,ARM64,DefaultCPU": "Follow build instructions from here", @@ -1005,7 +1005,7 @@ var validCombos = { "Follow build instructions from here", "android,Java,ARM64,DefaultCPU": - "Add a dependency on com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", + "Add a dependency on com.microsoft.onnxruntime:onnxruntime-android or com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", "android,C-API,ARM32,DefaultCPU": "Follow build instructions from here", @@ -1014,7 +1014,7 @@ var validCombos = { "Follow build instructions from here", "android,Java,ARM32,DefaultCPU": - "Add a dependency on com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", + "Add a dependency on com.microsoft.onnxruntime:onnxruntime-android or com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", "android,C-API,X86,DefaultCPU": "Follow build instructions from here", @@ -1023,7 +1023,7 @@ var validCombos = { "Follow build instructions from here", "android,Java,X86,DefaultCPU": - "Add a dependency on com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", + "Add a dependency on com.microsoft.onnxruntime:onnxruntime-android or com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", "android,C-API,X64,DefaultCPU": "Follow build instructions from here", @@ -1032,7 +1032,7 @@ var validCombos = { "Follow build instructions from here", "android,Java,X64,DefaultCPU": - "Add a dependency on com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", + "Add a dependency on com.microsoft.onnxruntime:onnxruntime-android or com.microsoft.onnxruntime:onnxruntime-mobile using Maven/Gradle and refer to the mobile deployment guide", "android,C#,ARM64,DefaultCPU": "Install Nuget package Microsoft.ML.OnnxRuntime.
Refer to sample here.", @@ -1041,22 +1041,22 @@ var validCombos = { "Install Nuget package Microsoft.ML.OnnxRuntime.
Refer to sample here.", "ios,C-API,ARM64,DefaultCPU": - "Add 'onnxruntime-mobile-c' using CocoaPods and refer to the mobile deployment guide", + "Add 'onnxruntime-c' or 'onnxruntime-mobile-c' using CocoaPods and refer to the mobile deployment guide", "ios,C++,ARM64,DefaultCPU": - "Add 'onnxruntime-mobile-c' using CocoaPods and refer to the mobile deployment guide", + "Add 'onnxruntime-c' or 'onnxruntime-mobile-c' using CocoaPods and refer to the mobile deployment guide", "ios,C-API,ARM64,CoreML": - "Add 'onnxruntime-mobile-c' using CocoaPods and refer to the mobile deployment guide", + "Add 'onnxruntime-c' or 'onnxruntime-mobile-c' using CocoaPods and refer to the mobile deployment guide", "ios,C++,ARM64,CoreML": - "Add 'onnxruntime-mobile-c' using CocoaPods and refer to the mobile deployment guide", + "Add 'onnxruntime-c' or 'onnxruntime-mobile-c' using CocoaPods and refer to the mobile deployment guide", "ios,objectivec,ARM64,DefaultCPU": - "Add 'onnxruntime-mobile-objc' using CocoaPods and refer to the mobile deployment guide", + "Add 'onnxruntime-objc' or 'onnxruntime-mobile-objc' using CocoaPods and refer to the mobile deployment guide", "ios,objectivec,ARM64,CoreML": - "Add 'onnxruntime-mobile-objc' using CocoaPods and refer to the mobile deployment guide", + "Add 'onnxruntime-objc' or 'onnxruntime-mobile-objc' using CocoaPods and refer to the mobile deployment guide", "ios,C#,ARM64,DefaultCPU": "Install Nuget package Microsoft.ML.OnnxRuntime.
Refer to sample here.",