mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-14 18:12:05 +00:00
3681048474
### Description Staged: https://faxu.github.io/onnxruntime/docs/performance/ Main changes: - Restructure performance section to break into sub-categories - Move CUDA specific perf tuning tips to [CUDA EP page](https://faxu.github.io/onnxruntime/docs/execution-providers/CUDA-ExecutionProvider.html#performance-tuning) - Update [Transformer optimizer page](https://faxu.github.io/onnxruntime/docs/performance/transformers-optimization.html) to remove version-specific content... will be supported along with https://github.com/microsoft/onnxruntime/pull/14964 - Fix links to point to new pages
91 lines
4.4 KiB
Markdown
91 lines
4.4 KiB
Markdown
---
|
|
title: XNNPACK
|
|
description: Instructions to execute ONNX Runtime with the XNNPACK execution provider
|
|
parent: Execution Providers
|
|
nav_order: 9
|
|
---
|
|
{::options toc_levels="2" /}
|
|
|
|
# XNNPACK Execution Provider
|
|
|
|
Accelerate ONNX models on Android/iOS devices and WebAssembly with ONNX Runtime and the XNNPACK execution provider. [XNNPACK](https://github.com/google/XNNPACK) is a highly optimized library of floating-point neural network inference operators for ARM, WebAssembly, and x86 platforms.
|
|
|
|
## Contents
|
|
{: .no_toc }
|
|
|
|
* TOC placeholder
|
|
{:toc}
|
|
|
|
|
|
## Install
|
|
Pre-built packages of ONNX Runtime ([`onnxruntime-android`](https://mvnrepository.com/artifact/com.microsoft.onnxruntime/onnxruntime-android)) with XNNPACK EP for Android are published on Maven.
|
|
See [here](../install/index.md#install-on-android) for installation instructions.
|
|
|
|
Pre-built binaries(`onnxruntime-objc` and `onnxruntime-c`) of ONNX Runtime with XNNPACK EP for iOS are published to CocoaPods.
|
|
See [here](../install/index.md#install-on-ios) for installation instructions.
|
|
|
|
## Build
|
|
|
|
Please see the [Build page](../build/eps.md#xnnpack) for instructions on building a package that includes the XNNPACK EP.
|
|
|
|
Support build for Android/iOS/Windows/Linux
|
|
|
|
## Usage
|
|
|
|
The ONNX Runtime API details are [here](../api).
|
|
|
|
The XNNPACK EP can be used via the C, C++ or Java APIs
|
|
|
|
The XNNPACK EP must be explicitly registered when creating the inference session. For example:
|
|
|
|
```C++
|
|
Ort::Env env = Ort::Env{ORT_LOGGING_LEVEL_ERROR, "Default"};
|
|
Ort::SessionOptions so;
|
|
so.AppendExecutionProvider("XNNPACK", {{"intra_op_num_threads", std::to_string(intra_op_num_threads)}});
|
|
Ort::Session session(env, model_path, so);
|
|
```
|
|
|
|
## Configuration Options
|
|
|
|
### Recommended configuration
|
|
XNNPACK has a separate internal threadpool which can lead to contention with the ORT intra-op threadpool.
|
|
To minimize this, we recommend setting the following options:
|
|
1. Disable the ORT intra-op thread-pool spinning by adding the following to the session options:
|
|
```C++
|
|
so.AddConfigEntry(kOrtSessionOptionsConfigAllowIntraOpSpinning, "0");
|
|
```
|
|
2. Set the XNNPACK intra-op thread-pool size when registering the XNNPACK EP. The suggested value would be the number of physical cores on the device.
|
|
```C++
|
|
so.AppendExecutionProvider("XNNPACK", {{"intra_op_num_threads", std::to_string(intra_op_num_threads)}});
|
|
```
|
|
3. Set the ORT intra-op thread-pool size to 1:
|
|
```C++
|
|
so.SetIntraOpNumThreads(1);
|
|
```
|
|
|
|
This configuration will work well if your model is using XNNPACK for the nodes performing the compute-intensive work, as these operators are likely to use the intra-op threadpool. e.g. Conv, Gemm, MatMul operators.
|
|
|
|
If your model contains compute-intensive nodes using operators that are not currently supported by the XNNPACK EP these will be handled by the CPU EP. In that case better performance may be achieved by increasing the size of the ORT intra-op threadpool and potentially re-enabling spinning. Performance testing is the best way to determine the optimal configuration for your model.
|
|
### Available Options
|
|
##### intra_op_num_threads
|
|
|
|
The number of threads to use for the XNNPACK EP's internal intra-op thread-pool. This is the number of threads used to parallelize the execution within a node. The default value is 1. The value should be >= 1.
|
|
|
|
|
|
## Supported ops
|
|
Following ops are supported by the XNNPACK Execution Provider,
|
|
|
|
|Operator|Note||
|
|
|--------|------|-----|
|
|
|ai.onnx:AveragePool|Only 2D Pool is supported.|
|
|
|ai.onnx:Conv|Only 2D Conv is supported.<br/>Weights and bias should be constant.|
|
|
|ai.onnx:ConvTranspose|Only 2D ConvTranspose is supported.<br/>Weights and bias should be constant.|since 1.14|
|
|
|ai.onnx:MaxPool|Only 2D Pool is supported.|
|
|
|ai.onnx:Softmax|all opset below 13 is supported, only support opset 13 when AXIS is the last dimension|
|
|
|ai.onnx:QLinearConv|Only 2D Conv is supported.<br/>Weights and bias should be constant.<br/>All quantization scales and zero points should be constant.|
|
|
|ai.onnx:Resize|2D/4D Resize in `Bilinear mode` are supported|since 1.14|
|
|
|ai.onnx:Gemm|Only 2D Op is supported|since 1.14|
|
|
|ai.onnx:Matmul|Only 2D Op is supported|since 1.14|
|
|
|com.microsoft:QLinearAveragePool|Only 2D Pool is supported.<br/>All quantization scales and zero points should be constant.|
|
|
|com.microsoft:QLinearSoftmax|All quantization scales and zero points should be constant.|
|
|
|com.microsoft:QLinearConvTranspose|All quantization scales and zero points should be constant.|since 1.14|
|