mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-13 18:08:13 +00:00
Document Affinity & CloudEP (#14137)
This commit is contained in:
RandySheriffH
GitHub
parent
4940d33f3b
commit
1391f81c3f
8 changed files with 180 additions and 5 deletions
29
docs/build/eps.md
vendored
29
docs/build/eps.md
vendored
|
|
@ -817,3 +817,32 @@ See more information on the CANN Execution Provider [here](../execution-provider
|
|||
|
||||
* The CANN execution provider supports building for both x64 and aarch64 architectures.
|
||||
* CANN excution provider now is only supported on Linux.
|
||||
|
||||
## Azure
|
||||
|
||||
See the [Azure Execution Provider](../execution-providers/Azure-ExecutionProvider.md) documentation for more details.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
For Linux, before building, please:
|
||||
|
||||
* install openssl dev package into the system, which is openssl-dev for redhat and libssl-dev for ubuntu.
|
||||
* if have multiple openssl dev versions installed in the system, please set environment variable "OPENSSL_ROOT_DIR" to the desired version, for example:
|
||||
|
||||
```base
|
||||
set OPENSSL_ROOT_DIR=/usr/local/ssl3.x/
|
||||
```
|
||||
|
||||
### Build Instructions
|
||||
|
||||
#### Windows
|
||||
|
||||
```dos
|
||||
build.bat --config <Release|Debug|RelWithDebInfo> --build_shared_lib --build_wheel --use_azure
|
||||
```
|
||||
|
||||
#### Linux
|
||||
|
||||
```bash
|
||||
./build.sh --config <Release|Debug|RelWithDebInfo> --build_shared_lib --build_wheel --use_azure
|
||||
```
|
||||
|
|
|
|||
73
docs/execution-providers/Azure-ExecutionProvider.md
Normal file
73
docs/execution-providers/Azure-ExecutionProvider.md
Normal file
|
|
@ -0,0 +1,73 @@
|
|||
---
|
||||
title: Cloud - Azure
|
||||
description: Instructions to infer an ONNX model remotely with an Azure endpoint
|
||||
parent: Execution Providers
|
||||
nav_order: 11
|
||||
---
|
||||
|
||||
# Azure Execution Provider (Preview)
|
||||
|
||||
The Azure Execution Provider enables ONNX Runtime to invoke an remote Azure endpoint for inferenece, the endpoint must be deployed beforehand.
|
||||
To consume the endpoint, a model of same inputs and outputs must be loaded locally in the first place.
|
||||
|
||||
One use case for Azure Execution Provider is small-big models. E.g. A smaller model deployed on edge device for faster inference,
|
||||
while a bigger model deployed on Azure for higher precision, with Azure Execution Provider, a switch between the two could be easily achieved.
|
||||
Again, the two models must have same inputs and outputs.
|
||||
|
||||
Azure Execution Provider is in preview stage, all API(s) and usage are subjuct to change.
|
||||
|
||||
## Limitations
|
||||
|
||||
So far, Azure Execution Provider is limited to:
|
||||
* only support [triton](https://github.com/triton-inference-server) server on [AML](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-with-triton?tabs=python%2Cendpoint).
|
||||
* only build and run on Windows and Linux.
|
||||
* available only as python package, but user could also build from source and consume the feature by C/C++ API(s).
|
||||
|
||||
## Requirements
|
||||
|
||||
For Windows, please install [zlib](https://zlib.net/) and [re2](https://github.com/google/re2), and add their binaries into the system path.
|
||||
If built from source, zlib and re2 binaries could be easily located with:
|
||||
|
||||
```dos
|
||||
cd <build_output_path>
|
||||
dir /s zlib1.dll re2.dll
|
||||
```
|
||||
|
||||
For Linux, please make sure openssl is installed.
|
||||
|
||||
## Known Issue
|
||||
|
||||
For certain ubuntu versions, https call made by AzureEP might report error - "error setting certificate verify location ...".
|
||||
To silence it, please create file "/etc/pki/tls/certs/ca-bundles.crt" that link to "/etc/ssl/certs/ca-certificates.crt".
|
||||
|
||||
## Build
|
||||
|
||||
For build instructions, please see the [BUILD page](../build/eps.md#azure).
|
||||
|
||||
## Usage
|
||||
|
||||
### Python
|
||||
|
||||
```python
|
||||
from onnxruntime import *
|
||||
import numpy as np
|
||||
import os
|
||||
|
||||
sess_opt = SessionOptions()
|
||||
sess_opt.add_session_config_entry('azure.endpoint_type', 'triton'); # only support triton server for now
|
||||
sess_opt.add_session_config_entry('azure.uri', 'https://...')
|
||||
sess_opt.add_session_config_entry('azure.model_name', 'a_simple_model');
|
||||
sess_opt.add_session_config_entry('azure.model_version', '1'); # optional, default 1
|
||||
sess_opt.add_session_config_entry('azure.verbose', 'true'); # optional, default false
|
||||
|
||||
sess = InferenceSession('a_simple_model.onnx', sess_opt, providers=['CPUExecutionProvider','azureExecutionProvider'])
|
||||
|
||||
run_opt = RunOptions()
|
||||
run_opt.add_run_config_entry('use_azure', '1') # optional, default '0' to run inference locally.
|
||||
run_opt.add_run_config_entry('azure.auth_key', '...') # optional, required only when use_azure set to 1
|
||||
|
||||
x = np.array([1,2,3,4]).astype(np.float32)
|
||||
y = np.array([4,3,2,1]).astype(np.float32)
|
||||
|
||||
z = sess.run(None, {'X':x, 'Y':y}, run_opt)[0]
|
||||
```
|
||||
|
|
@ -2,7 +2,7 @@
|
|||
title: Add a new provider
|
||||
description: Instructions to add a new execution provider to ONNX Runtime
|
||||
parent: Execution Providers
|
||||
nav_order: 12
|
||||
nav_order: 13
|
||||
redirect_from: /docs/how-to/add-execution-provider
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
title: Community-maintained
|
||||
parent: Execution Providers
|
||||
has_children: true
|
||||
nav_order: 11
|
||||
nav_order: 12
|
||||
---
|
||||
# Community-maintained Providers
|
||||
This list of providers for specialized hardware is contributed and maintained by ONNX Runtime community partners.
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ ONNX Runtime supports many different execution providers today. Some of the EPs
|
|||
|Default CPU|[NVIDIA CUDA](../execution-providers/CUDA-ExecutionProvider.md)|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[Rockchip NPU](../execution-providers/community-maintained/RKNPU-ExecutionProvider.md) (*preview*)|
|
||||
|[Intel DNNL](../execution-providers/oneDNN-ExecutionProvider.md)|[NVIDIA TensorRT](../execution-providers/TensorRT-ExecutionProvider.md)|[ARM Compute Library](../execution-providers/community-maintained/ACL-ExecutionProvider.md) (*preview*)|[Xilinx Vitis-AI](../execution-providers/community-maintained/Vitis-AI-ExecutionProvider.md) (*preview*)|
|
||||
|[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)|[DirectML](../execution-providers/DirectML-ExecutionProvider.md)|[Android Neural Networks API](../execution-providers/NNAPI-ExecutionProvider.md)|[Huawei CANN](../execution-providers/community-maintained/CANN-ExecutionProvider.md) (*preview*)|
|
||||
|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[AMD MIGraphX](../execution-providers/community-maintained/MIGraphX-ExecutionProvider.md) (*preview*)|[ARM-NN](../execution-providers/community-maintained/ArmNN-ExecutionProvider.md) (*preview*)|
|
||||
|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[AMD MIGraphX](../execution-providers/community-maintained/MIGraphX-ExecutionProvider.md) (*preview*)|[ARM-NN](../execution-providers/community-maintained/ArmNN-ExecutionProvider.md) (*preview*)|[Azure](../execution-providers/Azure-ExecutionProvider.md) (*preview*)|
|
||||
||[AMD ROCm](../execution-providers/ROCm-ExecutionProvider.md) (*preview*)|[CoreML](../execution-providers/CoreML-ExecutionProvider.md) (*preview*)|
|
||||
||[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)|[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)|
|
||||
||[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[Qualcomm SNPE](../execution-providers/SNPE-ExecutionProvider.md)|
|
||||
|
|
|
|||
|
|
@ -169,7 +169,70 @@ Currently, there are no special provisions to employ mimalloc on Linux. It is re
|
|||
|
||||
### Thread management
|
||||
|
||||
* Use the appropriate ORT API to set intra and inter op num threads. Inter op num threads is only used when parallel execution is enabled.
|
||||
#### Set number of intra-op threads
|
||||
|
||||
Onnxruntime sessions utilize multi-threading to parallelize computation inside each operator.
|
||||
Customer could configure the number of threads like:
|
||||
|
||||
```python
|
||||
sess_opt = SessionOptions()
|
||||
sess_opt.intra_op_num_threads = 3
|
||||
sess = ort.InferenceSession('model.onnx', sess_opt)
|
||||
```
|
||||
|
||||
With above configuration, two threads will be created in the pool, so along with the main calling thread, there will be three threads in total to participate in intra-op computation.
|
||||
By default, each session will create one thread per phyical core (except the 1st core) and attach the thread to that core.
|
||||
However, if customer explicitly set the number of threads like showcased above, there will be no affinity set to any of the created thread.
|
||||
|
||||
In addition, Onnxruntime also allow customers to create a global intra-op thread pool to prevent overheated contentions among session thread pools, please find its usage [here](https://github.com/microsoft/onnxruntime/blob/68b5b2d7d33b6aa2d2b5cf8d89befb4a76e8e7d8/onnxruntime/test/global_thread_pools/test_main.cc#L98).
|
||||
|
||||
#### Set number of inter-op threads
|
||||
|
||||
A inter-op thread pool is for parallelism between operators, and will only be created when session execution mode set to parallel:
|
||||
|
||||
```python
|
||||
sess_opt = SessionOptions()
|
||||
sess_opt.execution_mode = ExecutionMode.ORT_PARALLEL
|
||||
sess_opt.inter_op_num_threads = 3
|
||||
sess = ort.InferenceSession('model.onnx', sess_opt)
|
||||
```
|
||||
|
||||
By default, inter-op thread pool will also have one thread per physical core.
|
||||
|
||||
#### Set intra-op thread affinity
|
||||
|
||||
For certain scenarios, it may be beneficial to customize intra-op thread affinities, for example:
|
||||
* There are multiple sessions run in parallel, customer might prefer their intra-op thread pools run on separate cores to avoid contention.
|
||||
* Customer want to limit a intra-op thread pool to run on only one of the NUMA nodes to reduce overhead of expensive cache miss among nodes.
|
||||
|
||||
For session intra-op thread pool, please read the [configuration](https://github.com/microsoft/onnxruntime/blob/68b5b2d7d33b6aa2d2b5cf8d89befb4a76e8e7d8/include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h#L180) and consume it like:
|
||||
|
||||
```python
|
||||
sess_opt = SessionOptions()
|
||||
sess_opt.intra_op_num_threads = 3
|
||||
sess_opt.add_session_config_entry('session.intra_op_thread_affinities', '1;2')
|
||||
sess = ort.InferenceSession('model.onnx', sess_opt, ...)
|
||||
```
|
||||
|
||||
For global thread pool, please read the [API](https://github.com/microsoft/onnxruntime/blob/68b5b2d7d33b6aa2d2b5cf8d89befb4a76e8e7d8/include/onnxruntime/core/session/onnxruntime_c_api.h#L3636) and [usage](https://github.com/microsoft/onnxruntime/blob/68b5b2d7d33b6aa2d2b5cf8d89befb4a76e8e7d8/onnxruntime/test/global_thread_pools/test_main.cc#L98).
|
||||
|
||||
#### Numa support and performance tuning
|
||||
|
||||
Since release 1.14, Onnxruntime thread pool could utilize all physical cores that are available over NUMA nodes.
|
||||
The intra-op thread pool will create a thread on every physical core (except the 1st core). E.g. assume there is a system of 2 NUMA nodes, each has 24 cores.
|
||||
Hence intra-op thread pool will create 47 threads, and set thread affinity to each core.
|
||||
|
||||
For NUMA systems, it is recommended to test a few thread settings to explore for best performance, in that threads allocated among NUMA nodes might has higher cache-miss overhead when cooperating with each other. For example, when number of intra-op threads has to be 8, there are different ways to set affinity:
|
||||
|
||||
```
|
||||
sess_opt = SessionOptions()
|
||||
sess_opt.intra_op_num_threads = 8
|
||||
sess_opt.add_session_config_entry('session.intra_op_thread_affinities', '3,4;5,6;7,8;9,10;11,12;13,14;15,16') # set affinities of all 7 threads to cores in the first NUMA node
|
||||
# sess_opt.add_session_config_entry('session.intra_op_thread_affinities', '3,4;5,6;7,8;9,10;49,50;51,52;53,54') # set affinities for first 4 threads to the first NUMA node, and others to the second
|
||||
sess = ort.InferenceSession('resnet50.onnx', sess_opt, ...)
|
||||
```
|
||||
|
||||
Test showed that setting affinities to a single NUMA node has nearly 20 percent performance improvement aginst the other case.
|
||||
|
||||
#### Custom threading callbacks
|
||||
|
||||
|
|
|
|||
|
|
@ -234,7 +234,9 @@
|
|||
<div class="col-lg-3 col-md-3 r-option version" role="option" tabindex="-1" aria-selected="false" id="ACL">
|
||||
<span>ACL (Preview)</span></div>
|
||||
<div class="col-lg-3 col-md-3 r-option version" role="option" tabindex="-1" aria-selected="false" id="ArmNN">
|
||||
<span>ArmNN (Preview)</span></div>
|
||||
<span>ArmNN (Preview)</span></div>
|
||||
<div class="col-lg-3 col-md-3 r-option version" role="option" tabindex="-1" aria-selected="false" id="Azure">
|
||||
<span>Azure (Preview)</span></div>
|
||||
<div class="col-lg-3 col-md-3 r-option version" role="option" tabindex="-1" aria-selected="false" id="CANN">
|
||||
<span>CANN (Preview)</span></div>
|
||||
<div class="col-lg-3 col-md-3 r-option version" role="option" tabindex="-1" aria-selected="false" id="MIGraphX">
|
||||
|
|
@ -249,6 +251,8 @@
|
|||
<span>Vitis AI (Preview)</span></div>
|
||||
<div class="col-lg-3 col-md-3 r-option version" role="option" tabindex="-1" aria-selected="false" id="XNNPACK">
|
||||
<span>XNNPACK (Preview)</span></div>
|
||||
<div class="col-lg-3 col-md-3 r-option version" role="option" tabindex="-1" aria-selected="false" id="Azure">
|
||||
<span>Azure (Preview)</span></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
|
|
|||
|
|
@ -1223,6 +1223,12 @@ var validCombos = {
|
|||
|
||||
"linux,C++,X64,CANN":
|
||||
"Follow build instructions from <a href='http://www.onnxruntime.ai/docs/execution-providers/community-maintained/CANN-ExecutionProvider.html#build' target='_blank'>here</a>.",
|
||||
|
||||
"windows,Python,X64,Azure":
|
||||
"Follow build instructions from <a href='https://aka.ms/build-ort-azure' target='_blank'>here</a>",
|
||||
|
||||
"linux,Python,X64,Azure":
|
||||
"Follow build instructions from <a href='https://aka.ms/build-ort-azure' target='_blank'>here</a>",
|
||||
};
|
||||
|
||||
function commandMessage(key) {
|
||||
|
|
|
|||
Loading…
Reference in a new issue