The ORT format is the format supported by reduced size ONNX Runtime builds. Reduced size builds may be more appropriate for use in size-constrained environments such as mobile and web applications.
Both ORT format models and ONNX models are supported by a full ONNX Runtime build.
Generally, the goal is that a particular version of ONNX Runtime can run models at the current (at time of the ONNX Runtime release) or older versions of the ORT format.
Though we try to maintain backwards compatibility, there have been some breaking changes.
ONNX Runtime version|ORT format version support|Notes
-|-|-
1.14+|v5, v4 (limited support)| See [here](https://github.com/microsoft/onnxruntime/blob/rel-1.14.0/docs/ORT_Format_Update_in_1.13.md#onnx-runtime-114) for details about limited v4 support.
ONNX models are converted to ORT format using the `convert_onnx_models_to_ort` script.
The conversion script performs two functions:
1. Loads and optimizes ONNX format models, and saves them in ORT format
2. Determines the operators, and optionally data types, required by the optimized models, and saves these in a configuration file for use in a reduced operator build, if required
The conversion script can run on a single ONNX model, or a directory. If run against a directory, the directory will be recursively searched for '.onnx' files to convert.
Each '.onnx' file is loaded, optimized, and saved in ORT format as a file with the '.ort' extension in the same location as the original '.onnx' file.
2. A [build configuration file](../../reference/operators/reduced-operator-config-file.md) ('required_operators.config') with the operators required by the optimized ONNX models.
If [type reduction](#enable-type-reduction) is enabled (ONNX Runtime version 1.7 or later) the configuration file will also include the required types for each operator, and is called 'required_operators_and_types.config'.
If you are using a pre-built ONNX Runtime [iOS](../../install/index.md#install-on-ios), [Android](../../install/index.md#install-on-android) or [web](../../install/index.md#javascript-installs) package, the build configuration file is not used and can be ignored.
The ORT model format is supported by version 1.5.2 of ONNX Runtime or later.
Conversion of ONNX format models to ORT format utilizes the ONNX Runtime python package, as the model is loaded into ONNX Runtime and optimized as part of the conversion process.
For ONNX Runtime version 1.8 and later the conversion script is run directly from the ONNX Runtime python package.
For earlier versions, the conversion script is run from the local ONNX Runtime repository.
### Install ONNX Runtime
Install the onnxruntime python package from [https://pypi.org/project/onnxruntime/](https://pypi.org/project/onnxruntime/) in order to convert models from ONNX format to the internal ORT format. Version 1.5.3 or higher is required.
#### Install the latest release
```bash
pip install onnxruntime
```
#### Install a previous release
If you are building ONNX Runtime from source (custom, reduced or minimal builds), you must match the python package version to the branch of the ONNX Runtime repository you checked out.
Convert the ONNX format model/s in the provided directory to ORT format models. All files with a `.onnx` extension will be processed. For each one, an ORT format model will be created in the same directory. A configuration file will also be created containing the list of required
operators for all converted models. This configuration file should be used as input to the minimal build via the `--include_ops_by_config` parameter.
positional arguments:
model_path_or_dir Provide path to ONNX model or directory containing ONNX model/s to convert. All files with a .onnx extension, including those in subdirectories, will be processed.
Style of optimization to perform on the ORT format model. Multiple values may be provided. The conversion will run once for each value. The general guidance is to use models optimized with 'Runtime' style when using NNAPI or CoreML and 'Fixed' style otherwise.
'Fixed': Run optimizations directly before saving the ORT format model. This bakes in any platform-specific optimizations. 'Runtime': Run basic optimizations directly and save certain other optimizations to be applied at runtime if possible. This is useful when
using a compiling EP like NNAPI or CoreML that may run an unknown (at model conversion time) number of nodes. The saved optimizations can further optimize nodes not assigned to the compiling EP at runtime.
--enable_type_reduction
Add operator specific type information to the configuration file to potentially reduce the types supported by individual operator implementations.
--custom_op_library CUSTOM_OP_LIBRARY
Provide path to shared library containing custom operator kernels to register.
--save_optimized_onnx_model
Save the optimized version of each ONNX model. This will have the same level of optimizations applied as the ORT format model.
--allow_conversion_failures
Whether to proceed after encountering model conversion failures.
Specify the list of NNAPI EP partitioning stop ops. In particular, specify the value of the "ep.nnapi.partitioning_stop_ops" session options config entry.
--target_platform {arm,amd64}
Specify the target platform where the exported model will be used. This parameter can be used to choose between platform-specific options, such as QDQIsInt8Allowed(arm), NCHWc (amd64) and NHWC (arm/amd64) format, different optimizer level options, etc.
Specify whether the converted model will be fully optimized ("Fixed") or have saved runtime optimizations ("Runtime"). Both types of models are produced by default. See [here](./ort-format-model-runtime-optimization.md) for more information.
For earlier versions, *extended* is recommended, as the *all* level previously included device specific optimizations that would limit the portability of the model.
If the model is to be run with the NNAPI EP or CoreML EP, it is recommended to create an ORT format model using the *basic* optimization level. Performance testing should be done to compare running this model with the NNAPI or CoreML EP enabled vs. running the model optimized to a higher level using the CPU EP to determine the optimal setup.
With ONNX Runtime version 1.7 and later it is possible to limit the data types the required operators support to further reduce the build size. This pruning is referred to as "operator type reduction" in this documentation. As the ONNX model/s are converted, the input and output data types required by each operator are accumulated and included in the configuration file.
If you wish to enable operator type reduction, the [Flatbuffers](https://google.github.io/flatbuffers/) python package must be installed.
```bash
pip install flatbuffers
```
For example, the ONNX Runtime kernel for Softmax supports both float and double. If your model/s uses Softmax but only with float data, we can exclude the implementation that supports double to reduce the kernel's binary size.
If your ONNX model uses [custom operators](../../reference/operators/add-custom-op.md), the path to the library containing the custom operator kernels must be provided so that the ONNX model can be successfully loaded. The custom operators will be preserved in the ORT format model.
Add this flag to save the optimized ONNX model. The optimized ONNX model contains the same nodes and initializers as the ORT format model, and can be viewed in [Netron](https://netron.app/) for debugging and performance tuning.
### Previous versions of ONNX Runtime
Prior to ONNX Runtime version 1.7, the model conversion script must be run from a cloned source repository:
If you wish to explicitly say that the InferenceSession input is an ORT format model you can do so via SessionOptions, although this generally should not be necessary.
#### Load ORT format model from an in-memory byte array
If a session is created using an input byte array containing the ORT format model data, by default we will copy the model bytes at the time of session creation to ensure the model bytes buffer is valid.
You may also enable the option to use the model bytes directly by setting the SessionOptions config entry `session.use_ort_model_bytes_directly` to `1`. This may reduce the peak memory usage of ONNX Runtime Mobile, but you will need to guarantee that the model bytes are valid throughout the lifespan of the ORT session. For ONNX Runtime Web, this option is set by default.
If `session.use_ort_model_bytes_directly` is enabled there is also an option to directly use the model bytes for initializers to further reduce peak memory usage. Set the Session Options config entry `session.use_ort_model_bytes_for_initializers` to `1` to enable this.
Note that if an initializer gets pre-packed it will undo the peak memory usage saving from using the model bytes directly for that initializer, as a new buffer for the pre-packed data needs to be allocated. Pre-packing is an optional performance optimization that involves changing the initializer layout to the optimal ordering for the current platform if it differs. If reducing peak memory usage is more important than potential performance optimizations pre-packing can be disabled by setting `session.disable_prepacking` to `1`.
