diff --git a/docs/build/eps.md b/docs/build/eps.md index d3e17007c1..4d56072038 100644 --- a/docs/build/eps.md +++ b/docs/build/eps.md @@ -410,30 +410,38 @@ Dockerfile instructions are available [here](https://github.com/microsoft/onnxru --- -## SNPE -See more information on the SNPE execution provider [here](../execution-providers/SNPE-ExecutionProvider.md). +## QNN +See more information on the QNN execution provider [here](../execution-providers/QNN-ExecutionProvider.md). ### Prerequisites {: .no_toc } -* Qualcomm Neural Processing SDK [Linux/Android](https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk) -* Qualcomm Neural Processing SDK [Windows](https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk/windows-on-snapdragon) +* Qualcomm AI Engine Direct SDK (Qualcomm Neural Network SDK) [Linux/Android/Windows](https://qpm.qualcomm.com/main/tools/details/qualcomm_ai_engine_direct) ### Build Instructions {: .no_toc } -#### Windows +#### Windows (arm64 native build) ``` -build.bat --use_snpe --snpe_root=[location-of-SNPE_SDK] --build_shared_lib --cmake_generator "Visual Studio 16 2019" --skip_submodule_sync --config Release --build_dir \build\Windows +build.bat --arm64 --use_qnn --qnn_home=[QNN_SDK path] --build_shared_lib --cmake_generator "Visual Studio 17 2022" --skip_submodule_sync --config Release --build_dir \build\Windows +``` + +build python bindings +``` +build.bat --arm64 --use_qnn --qnn_home=[QNN_SDK path] --build_wheel --cmake_generator "Visual Studio 17 2022" --skip_submodule_sync --config Release --build_dir \build\Windows +``` +#### Linux (x64) +``` +build.py --use_qnn --qnn_home=[QNN_SDK path] --build_shared_lib --skip_submodule_sync --config Release ``` #### Android (Cross-Compile): Please reference [Build OnnxRuntime For Android](android.md) ``` # on Windows -build.bat --build_shared_lib --skip_submodule_sync --android --config Release --use_snpe --snpe_root [location-of-SNPE_SDK] --android_sdk_path [location-of-android_SDK] --android_ndk_path [location-of-android_NDK] --android_abi arm64-v8a --android_api [api-version] --cmake_generator Ninja --build_dir build\Android +build.bat --build_shared_lib --skip_submodule_sync --android --config Release --use_qnn --qnn_home [QNN_SDK path] --android_sdk_path [android_SDK path] --android_ndk_path [android_NDK path] --android_abi arm64-v8a --android_api [api-version] --cmake_generator Ninja --build_dir build\Android # on Linux -build.sh --build_shared_lib --skip_submodule_sync --android --config Release --use_snpe --snpe_root [location-of-SNPE_SDK] --android_sdk_path [location-of-android_SDK] --android_ndk_path [location-of-android_NDK] --android_abi arm64-v8a --android_api [api-version] --cmake_generator Ninja --build_dir build/Android +build.sh --build_shared_lib --skip_submodule_sync --android --config Release --use_qnn --qnn_home [QNN_SDK path] --android_sdk_path [android_SDK path] --android_ndk_path [android_NDK path] --android_abi arm64-v8a --android_api [api-version] --cmake_generator Ninja --build_dir build/Android ``` diff --git a/docs/execution-providers/QNN-ExecutionProvider.md b/docs/execution-providers/QNN-ExecutionProvider.md new file mode 100644 index 0000000000..3cbb8564af --- /dev/null +++ b/docs/execution-providers/QNN-ExecutionProvider.md @@ -0,0 +1,96 @@ +--- +title: Qualcomm - QNN +description: Execute ONNX models with QNN Execution Provider +parent: Execution Providers +nav_order: 6 +redirect_from: /docs/reference/execution-providers/QNN-ExecutionProvider +--- + +# QNN Execution Provider +{: .no_toc } + +The QNN Execution Provider for ONNX Runtime enables hardware accelerated execution on Qualcomm chipsets. +It uses the Qualcomm AI Engine Direct SDK (QNN SDK) to construct a QNN graph from an ONNX model which can +be executed by a supported accelerator backend library. + + +## Contents +{: .no_toc } + +* TOC placeholder +{:toc} + +## Install Pre-requisites + +Download the Qualcomm AI Engine Direct SDK (QNN SDK) from [https://qpm.qualcomm.com/main/tools/details/qualcomm_ai_engine_direct](https://qpm.qualcomm.com/main/tools/details/qualcomm_ai_engine_direct) + +### QNN Version Requirements + +ONNX Runtime QNN Execution Provider has been built and tested with QNN 2.10.x and Qualcomm SC8280, SM8350 SOC's + +## Build +For build instructions, please see the [BUILD page](../build/eps.md#qnn). + +## Configuration Options +The QNN Execution Provider supports a number of configuration options. The `provider_option_keys`, `provider_options_values` enable different options for the application. Each `provider_options_keys` accepts values as shown below: + +|`provider_options_values` for `provider_options_keys = "backend_path"`|Description| +|---|-----| +|'libQnnCpu.so' or 'QnnCpu.dll'|Enable CPU backend. Useful for integration testing. CPU backend is a reference implementation of QNN operators| +|'libQnnHtp.do' or 'QnnHtp.dll'|Enable Htp backend. Offloads compute to NPU.| + +|`provider_options_values` for `provider_options_keys = "profiling_level"`|Description| +|---|---| +|'off'|| +|'basic'|| +|'detailed'|| + +|`provider_options_values` for `provider_options_keys = "rpc_control_latency"`|Description| +|---|---| +|microseconds (string)|allows client to set up RPC control latency in microseconds| + +|`provider_options_values` for `provider_options_keys = "htp_performance_mode"`|Description| +|---|---| +|'burst'|| +|'balanced'|| +|'default'|| +|'high_performance'|| +|'high_power_saver'|| +|'low_balanced'|| +|'low_power_saver'|| +|'power_saver'|| +|'sustained_high_performance'|| + + +|`provider_options_values` for `provider_options_keys = "qnn_context_cache_enable"`|Description| +|---|---| +|'0'|disabled (default)| +|'1'|enable qnn context cache. write out prepared Htp Context Binary to disk to save initialization costs.| + + +|`provider_options_values` for `provider_options_keys = "qnn_context_cache_path"`|Description| +|---|---| +|'/path/to/context/cache'|string path to context cache binary| + + +## Usage +### C++ +C API details are [here](../get-started/with-c.md). +``` +Ort::Env env = Ort::Env{ORT_LOGGING_LEVEL_ERROR, "Default"}; +std::unordered_map qnn_options; +qnn_options["backend_path"] = "QnnHtp.dll"; +Ort::SessionOptions session_options; +session_options.AppendExecutionProvider("QNN", qnn_options); +Ort::Session session(env, model_path, session_options); +``` +### Python +``` +import onnxruntime as ort +# Create a session with QNN EP using HTP (NPU) backend. +sess = ort.InferenceSession(model_path, providers=['QNNExecutionProvider'], provider_options=[{'backend_path':'QnnHtp.dll'}])` +``` + +### Inference example + +[Image classification with SqueezeNet in CPP using QNN Execution Provider with QNN CPU Backend](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_cxx/QNN_EP) diff --git a/docs/execution-providers/SNPE-ExecutionProvider.md b/docs/execution-providers/SNPE-ExecutionProvider.md deleted file mode 100644 index f40c951df3..0000000000 --- a/docs/execution-providers/SNPE-ExecutionProvider.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -title: Qualcomm - SNPE -description: Execute ONNX models with SNPE Execution Provider -parent: Execution Providers -nav_order: 6 -redirect_from: /docs/reference/execution-providers/SNPE-ExecutionProvider ---- - -# SNPE Execution Provider -{: .no_toc } - -The SNPE Execution Provider for ONNX Runtime enables hardware accelerated execution on Qualcomm Snapdragon CPU, the Qualcomm AdrenoTM GPU, or the Hexagon DSP. This execution provider makes use of the Qualcomm Snapdragon Neural Processing Engine SDK. - -This execution provider uses the AOT converted DLC code as an embedded node in the ONNX model file. - -## Contents -{: .no_toc } - -* TOC placeholder -{:toc} - -## Install Pre-requisites - -Download the SNPE toolkit from the Qualcomm Developer Network for [Android/Linux](https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk) -or [Windows](https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk/windows-on-snapdragon) - -### SNPE Version Requirements - -The SNPE version used with the ONNX Runtime SNPE Execution Provider must match the version used to generate the quantized SNPE-DLC file. - -## Build -For build instructions, please see the [BUILD page](../build/eps.md#snpe). - -## Configuration Options -The SNPE Execution Provider supports a number of options to set the SNPE Runtime configuration for executing the model. The `provider_option_keys`, `provider_options_values` and `num_keys` enable different options for the application. Each `provider_options_keys` accepts values as shown below: - -|`provider_options_values` for `provider_options_keys = "runtime"`|Description| -|---|-----| -|CPU or CPU_FLOAT32|Using SnapDragon CPU with 32 bit data storage and math| -|DSP or DSP_FIXED8_TF|Using Hexagon DSP with 8bit fixed point Tensorflow style format data storage and 8bit fixed point Tensorflow style format math| -|GPU or GPU_FLOAT32_16_HYBRID|Using Adreno GPU with 16 bit data storage and 32 bit math| -|GPU_FLOAT16|GPU with 16 bit data storage and 16 bit math| -|AIP_FIXED_TF or AIP_FIXED8_TF|Using Snapdragon AIX+HVX with 8bit fixed point Tensorflow style format data storage and 8bit fixed point Tensorflow style format math| - -|`provider_options_values` for `provider_options_keys = "buffer_type"`|Description| -|---|---| -|ITensor|Represents a tensor with n-dimensional data| -|TF8|User defined buffer with 8-bit quantized value| -|TF16|User defined buffer with 16-bit quantized value| -|UINT8|User defined buffer with unsigned int value| -|FLOAT|User defined buffer with float value| - -## Usage -### C++ -``` -Ort::Env env = Ort::Env{ORT_LOGGING_LEVEL_ERROR, "Default"}; -std::unordered_map snpe_options; -snpe_options["runtime"] = "DSP"; -snpe_options["buffer_type"] = "FLOAT"; -Ort::SessionOptions session_options; -session_options.AppendExecutionProvider("SNPE", snpe_options); -Ort::Session session(env, model_path, session_options); -``` - -The C API details are [here](../get-started/with-c.md). - -### Inference example - -[Image classification with Inception v3 in CPP using SNPE Execution Provider](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_cxx/Snpe_EP) - -[Image classification with VGG16 in C# using SNPE Execution Provider](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_sharp/Snpe_EP/vgg16_image_classification) diff --git a/docs/execution-providers/index.md b/docs/execution-providers/index.md index ef0c85fc6e..0e3488ca59 100644 --- a/docs/execution-providers/index.md +++ b/docs/execution-providers/index.md @@ -29,7 +29,7 @@ ONNX Runtime supports many different execution providers today. Some of the EPs |[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[AMD MIGraphX](../execution-providers/MIGraphX-ExecutionProvider.md)|[ARM-NN](../execution-providers/community-maintained/ArmNN-ExecutionProvider.md) (*preview*)| |[XNNPACK](../execution-providers/Xnnpack-ExecutionProvider.md)|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[CoreML](../execution-providers/CoreML-ExecutionProvider.md) (*preview*)| ||[AMD ROCm](../execution-providers/ROCm-ExecutionProvider.md)|[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)| -||[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)|[Qualcomm SNPE](../execution-providers/SNPE-ExecutionProvider.md)| +||[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)|[Qualcomm QNN](../execution-providers/QNN-ExecutionProvider.md)| |||[XNNPACK](../execution-providers/Xnnpack-ExecutionProvider.md)| ## Add an Execution Provider