mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-18 18:52:16 +00:00
Add initial QNN EP docs (#16053)
remove SNPE EP and add initial QNN EP docs. (more updates coming pending Qualcomm online docs)
This commit is contained in:
George Wu
GitHub
parent
8b82f5ec2d
commit
9e6613db1e
4 changed files with 113 additions and 80 deletions
24
docs/build/eps.md
vendored
24
docs/build/eps.md
vendored
|
|
@ -410,30 +410,38 @@ Dockerfile instructions are available [here](https://github.com/microsoft/onnxru
|
|||
|
||||
---
|
||||
|
||||
## SNPE
|
||||
See more information on the SNPE execution provider [here](../execution-providers/SNPE-ExecutionProvider.md).
|
||||
## QNN
|
||||
See more information on the QNN execution provider [here](../execution-providers/QNN-ExecutionProvider.md).
|
||||
|
||||
### Prerequisites
|
||||
{: .no_toc }
|
||||
* Qualcomm Neural Processing SDK [Linux/Android](https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk)
|
||||
* Qualcomm Neural Processing SDK [Windows](https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk/windows-on-snapdragon)
|
||||
* Qualcomm AI Engine Direct SDK (Qualcomm Neural Network SDK) [Linux/Android/Windows](https://qpm.qualcomm.com/main/tools/details/qualcomm_ai_engine_direct)
|
||||
|
||||
### Build Instructions
|
||||
{: .no_toc }
|
||||
|
||||
#### Windows
|
||||
#### Windows (arm64 native build)
|
||||
```
|
||||
build.bat --use_snpe --snpe_root=[location-of-SNPE_SDK] --build_shared_lib --cmake_generator "Visual Studio 16 2019" --skip_submodule_sync --config Release --build_dir \build\Windows
|
||||
build.bat --arm64 --use_qnn --qnn_home=[QNN_SDK path] --build_shared_lib --cmake_generator "Visual Studio 17 2022" --skip_submodule_sync --config Release --build_dir \build\Windows
|
||||
```
|
||||
|
||||
build python bindings
|
||||
```
|
||||
build.bat --arm64 --use_qnn --qnn_home=[QNN_SDK path] --build_wheel --cmake_generator "Visual Studio 17 2022" --skip_submodule_sync --config Release --build_dir \build\Windows
|
||||
```
|
||||
#### Linux (x64)
|
||||
```
|
||||
build.py --use_qnn --qnn_home=[QNN_SDK path] --build_shared_lib --skip_submodule_sync --config Release
|
||||
```
|
||||
#### Android (Cross-Compile):
|
||||
|
||||
Please reference [Build OnnxRuntime For Android](android.md)
|
||||
```
|
||||
# on Windows
|
||||
build.bat --build_shared_lib --skip_submodule_sync --android --config Release --use_snpe --snpe_root [location-of-SNPE_SDK] --android_sdk_path [location-of-android_SDK] --android_ndk_path [location-of-android_NDK] --android_abi arm64-v8a --android_api [api-version] --cmake_generator Ninja --build_dir build\Android
|
||||
build.bat --build_shared_lib --skip_submodule_sync --android --config Release --use_qnn --qnn_home [QNN_SDK path] --android_sdk_path [android_SDK path] --android_ndk_path [android_NDK path] --android_abi arm64-v8a --android_api [api-version] --cmake_generator Ninja --build_dir build\Android
|
||||
|
||||
# on Linux
|
||||
build.sh --build_shared_lib --skip_submodule_sync --android --config Release --use_snpe --snpe_root [location-of-SNPE_SDK] --android_sdk_path [location-of-android_SDK] --android_ndk_path [location-of-android_NDK] --android_abi arm64-v8a --android_api [api-version] --cmake_generator Ninja --build_dir build/Android
|
||||
build.sh --build_shared_lib --skip_submodule_sync --android --config Release --use_qnn --qnn_home [QNN_SDK path] --android_sdk_path [android_SDK path] --android_ndk_path [android_NDK path] --android_abi arm64-v8a --android_api [api-version] --cmake_generator Ninja --build_dir build/Android
|
||||
|
||||
```
|
||||
|
||||
|
|
|
|||
96
docs/execution-providers/QNN-ExecutionProvider.md
Normal file
96
docs/execution-providers/QNN-ExecutionProvider.md
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
---
|
||||
title: Qualcomm - QNN
|
||||
description: Execute ONNX models with QNN Execution Provider
|
||||
parent: Execution Providers
|
||||
nav_order: 6
|
||||
redirect_from: /docs/reference/execution-providers/QNN-ExecutionProvider
|
||||
---
|
||||
|
||||
# QNN Execution Provider
|
||||
{: .no_toc }
|
||||
|
||||
The QNN Execution Provider for ONNX Runtime enables hardware accelerated execution on Qualcomm chipsets.
|
||||
It uses the Qualcomm AI Engine Direct SDK (QNN SDK) to construct a QNN graph from an ONNX model which can
|
||||
be executed by a supported accelerator backend library.
|
||||
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Install Pre-requisites
|
||||
|
||||
Download the Qualcomm AI Engine Direct SDK (QNN SDK) from [https://qpm.qualcomm.com/main/tools/details/qualcomm_ai_engine_direct](https://qpm.qualcomm.com/main/tools/details/qualcomm_ai_engine_direct)
|
||||
|
||||
### QNN Version Requirements
|
||||
|
||||
ONNX Runtime QNN Execution Provider has been built and tested with QNN 2.10.x and Qualcomm SC8280, SM8350 SOC's
|
||||
|
||||
## Build
|
||||
For build instructions, please see the [BUILD page](../build/eps.md#qnn).
|
||||
|
||||
## Configuration Options
|
||||
The QNN Execution Provider supports a number of configuration options. The `provider_option_keys`, `provider_options_values` enable different options for the application. Each `provider_options_keys` accepts values as shown below:
|
||||
|
||||
|`provider_options_values` for `provider_options_keys = "backend_path"`|Description|
|
||||
|---|-----|
|
||||
|'libQnnCpu.so' or 'QnnCpu.dll'|Enable CPU backend. Useful for integration testing. CPU backend is a reference implementation of QNN operators|
|
||||
|'libQnnHtp.do' or 'QnnHtp.dll'|Enable Htp backend. Offloads compute to NPU.|
|
||||
|
||||
|`provider_options_values` for `provider_options_keys = "profiling_level"`|Description|
|
||||
|---|---|
|
||||
|'off'||
|
||||
|'basic'||
|
||||
|'detailed'||
|
||||
|
||||
|`provider_options_values` for `provider_options_keys = "rpc_control_latency"`|Description|
|
||||
|---|---|
|
||||
|microseconds (string)|allows client to set up RPC control latency in microseconds|
|
||||
|
||||
|`provider_options_values` for `provider_options_keys = "htp_performance_mode"`|Description|
|
||||
|---|---|
|
||||
|'burst'||
|
||||
|'balanced'||
|
||||
|'default'||
|
||||
|'high_performance'||
|
||||
|'high_power_saver'||
|
||||
|'low_balanced'||
|
||||
|'low_power_saver'||
|
||||
|'power_saver'||
|
||||
|'sustained_high_performance'||
|
||||
|
||||
|
||||
|`provider_options_values` for `provider_options_keys = "qnn_context_cache_enable"`|Description|
|
||||
|---|---|
|
||||
|'0'|disabled (default)|
|
||||
|'1'|enable qnn context cache. write out prepared Htp Context Binary to disk to save initialization costs.|
|
||||
|
||||
|
||||
|`provider_options_values` for `provider_options_keys = "qnn_context_cache_path"`|Description|
|
||||
|---|---|
|
||||
|'/path/to/context/cache'|string path to context cache binary|
|
||||
|
||||
|
||||
## Usage
|
||||
### C++
|
||||
C API details are [here](../get-started/with-c.md).
|
||||
```
|
||||
Ort::Env env = Ort::Env{ORT_LOGGING_LEVEL_ERROR, "Default"};
|
||||
std::unordered_map<std::string, std::string> qnn_options;
|
||||
qnn_options["backend_path"] = "QnnHtp.dll";
|
||||
Ort::SessionOptions session_options;
|
||||
session_options.AppendExecutionProvider("QNN", qnn_options);
|
||||
Ort::Session session(env, model_path, session_options);
|
||||
```
|
||||
### Python
|
||||
```
|
||||
import onnxruntime as ort
|
||||
# Create a session with QNN EP using HTP (NPU) backend.
|
||||
sess = ort.InferenceSession(model_path, providers=['QNNExecutionProvider'], provider_options=[{'backend_path':'QnnHtp.dll'}])`
|
||||
```
|
||||
|
||||
### Inference example
|
||||
|
||||
[Image classification with SqueezeNet in CPP using QNN Execution Provider with QNN CPU Backend](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_cxx/QNN_EP)
|
||||
|
|
@ -1,71 +0,0 @@
|
|||
---
|
||||
title: Qualcomm - SNPE
|
||||
description: Execute ONNX models with SNPE Execution Provider
|
||||
parent: Execution Providers
|
||||
nav_order: 6
|
||||
redirect_from: /docs/reference/execution-providers/SNPE-ExecutionProvider
|
||||
---
|
||||
|
||||
# SNPE Execution Provider
|
||||
{: .no_toc }
|
||||
|
||||
The SNPE Execution Provider for ONNX Runtime enables hardware accelerated execution on Qualcomm Snapdragon CPU, the Qualcomm Adreno<sup>TM</sup> GPU, or the Hexagon DSP. This execution provider makes use of the Qualcomm Snapdragon Neural Processing Engine SDK.
|
||||
|
||||
This execution provider uses the AOT converted DLC code as an embedded node in the ONNX model file.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
||||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
## Install Pre-requisites
|
||||
|
||||
Download the SNPE toolkit from the Qualcomm Developer Network for [Android/Linux](https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk)
|
||||
or [Windows](https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk/windows-on-snapdragon)
|
||||
|
||||
### SNPE Version Requirements
|
||||
|
||||
The SNPE version used with the ONNX Runtime SNPE Execution Provider must match the version used to generate the quantized SNPE-DLC file.
|
||||
|
||||
## Build
|
||||
For build instructions, please see the [BUILD page](../build/eps.md#snpe).
|
||||
|
||||
## Configuration Options
|
||||
The SNPE Execution Provider supports a number of options to set the SNPE Runtime configuration for executing the model. The `provider_option_keys`, `provider_options_values` and `num_keys` enable different options for the application. Each `provider_options_keys` accepts values as shown below:
|
||||
|
||||
|`provider_options_values` for `provider_options_keys = "runtime"`|Description|
|
||||
|---|-----|
|
||||
|CPU or CPU_FLOAT32|Using SnapDragon CPU with 32 bit data storage and math|
|
||||
|DSP or DSP_FIXED8_TF|Using Hexagon DSP with 8bit fixed point Tensorflow style format data storage and 8bit fixed point Tensorflow style format math|
|
||||
|GPU or GPU_FLOAT32_16_HYBRID|Using Adreno GPU with 16 bit data storage and 32 bit math|
|
||||
|GPU_FLOAT16|GPU with 16 bit data storage and 16 bit math|
|
||||
|AIP_FIXED_TF or AIP_FIXED8_TF|Using Snapdragon AIX+HVX with 8bit fixed point Tensorflow style format data storage and 8bit fixed point Tensorflow style format math|
|
||||
|
||||
|`provider_options_values` for `provider_options_keys = "buffer_type"`|Description|
|
||||
|---|---|
|
||||
|ITensor|Represents a tensor with n-dimensional data|
|
||||
|TF8|User defined buffer with 8-bit quantized value|
|
||||
|TF16|User defined buffer with 16-bit quantized value|
|
||||
|UINT8|User defined buffer with unsigned int value|
|
||||
|FLOAT|User defined buffer with float value|
|
||||
|
||||
## Usage
|
||||
### C++
|
||||
```
|
||||
Ort::Env env = Ort::Env{ORT_LOGGING_LEVEL_ERROR, "Default"};
|
||||
std::unordered_map<std::string, std::string> snpe_options;
|
||||
snpe_options["runtime"] = "DSP";
|
||||
snpe_options["buffer_type"] = "FLOAT";
|
||||
Ort::SessionOptions session_options;
|
||||
session_options.AppendExecutionProvider("SNPE", snpe_options);
|
||||
Ort::Session session(env, model_path, session_options);
|
||||
```
|
||||
|
||||
The C API details are [here](../get-started/with-c.md).
|
||||
|
||||
### Inference example
|
||||
|
||||
[Image classification with Inception v3 in CPP using SNPE Execution Provider](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_cxx/Snpe_EP)
|
||||
|
||||
[Image classification with VGG16 in C# using SNPE Execution Provider](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_sharp/Snpe_EP/vgg16_image_classification)
|
||||
|
|
@ -29,7 +29,7 @@ ONNX Runtime supports many different execution providers today. Some of the EPs
|
|||
|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[AMD MIGraphX](../execution-providers/MIGraphX-ExecutionProvider.md)|[ARM-NN](../execution-providers/community-maintained/ArmNN-ExecutionProvider.md) (*preview*)|
|
||||
|[XNNPACK](../execution-providers/Xnnpack-ExecutionProvider.md)|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[CoreML](../execution-providers/CoreML-ExecutionProvider.md) (*preview*)|
|
||||
||[AMD ROCm](../execution-providers/ROCm-ExecutionProvider.md)|[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)|
|
||||
||[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)|[Qualcomm SNPE](../execution-providers/SNPE-ExecutionProvider.md)|
|
||||
||[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)|[Qualcomm QNN](../execution-providers/QNN-ExecutionProvider.md)|
|
||||
|||[XNNPACK](../execution-providers/Xnnpack-ExecutionProvider.md)|
|
||||
|
||||
## Add an Execution Provider
|
||||
|
|
|
|||
Loading…
Reference in a new issue