diff --git a/docs/build/eps.md b/docs/build/eps.md index f429b74515..567f8d6420 100644 --- a/docs/build/eps.md +++ b/docs/build/eps.md @@ -817,3 +817,32 @@ See more information on the CANN Execution Provider [here](../execution-provider * The CANN execution provider supports building for both x64 and aarch64 architectures. * CANN excution provider now is only supported on Linux. + +## Azure + +See the [Azure Execution Provider](../execution-providers/Azure-ExecutionProvider.md) documentation for more details. + +### Prerequisites + +For Linux, before building, please: + +* install openssl dev package into the system, which is openssl-dev for redhat and libssl-dev for ubuntu. +* if have multiple openssl dev versions installed in the system, please set environment variable "OPENSSL_ROOT_DIR" to the desired version, for example: + +```base +set OPENSSL_ROOT_DIR=/usr/local/ssl3.x/ +``` + +### Build Instructions + +#### Windows + +```dos +build.bat --config --build_shared_lib --build_wheel --use_azure +``` + +#### Linux + +```bash +./build.sh --config --build_shared_lib --build_wheel --use_azure +``` diff --git a/docs/execution-providers/Azure-ExecutionProvider.md b/docs/execution-providers/Azure-ExecutionProvider.md new file mode 100644 index 0000000000..ff503ebcd5 --- /dev/null +++ b/docs/execution-providers/Azure-ExecutionProvider.md @@ -0,0 +1,73 @@ +--- +title: Cloud - Azure +description: Instructions to infer an ONNX model remotely with an Azure endpoint +parent: Execution Providers +nav_order: 11 +--- + +# Azure Execution Provider (Preview) + +The Azure Execution Provider enables ONNX Runtime to invoke an remote Azure endpoint for inferenece, the endpoint must be deployed beforehand. +To consume the endpoint, a model of same inputs and outputs must be loaded locally in the first place. + +One use case for Azure Execution Provider is small-big models. E.g. A smaller model deployed on edge device for faster inference, +while a bigger model deployed on Azure for higher precision, with Azure Execution Provider, a switch between the two could be easily achieved. +Again, the two models must have same inputs and outputs. + +Azure Execution Provider is in preview stage, all API(s) and usage are subjuct to change. + +## Limitations + +So far, Azure Execution Provider is limited to: +* only support [triton](https://github.com/triton-inference-server) server on [AML](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-with-triton?tabs=python%2Cendpoint). +* only build and run on Windows and Linux. +* available only as python package, but user could also build from source and consume the feature by C/C++ API(s). + +## Requirements + +For Windows, please install [zlib](https://zlib.net/) and [re2](https://github.com/google/re2), and add their binaries into the system path. +If built from source, zlib and re2 binaries could be easily located with: + +```dos +cd +dir /s zlib1.dll re2.dll +``` + +For Linux, please make sure openssl is installed. + +## Known Issue + +For certain ubuntu versions, https call made by AzureEP might report error - "error setting certificate verify location ...". +To silence it, please create file "/etc/pki/tls/certs/ca-bundles.crt" that link to "/etc/ssl/certs/ca-certificates.crt". + +## Build + +For build instructions, please see the [BUILD page](../build/eps.md#azure). + +## Usage + +### Python + +```python +from onnxruntime import * +import numpy as np +import os + +sess_opt = SessionOptions() +sess_opt.add_session_config_entry('azure.endpoint_type', 'triton'); # only support triton server for now +sess_opt.add_session_config_entry('azure.uri', 'https://...') +sess_opt.add_session_config_entry('azure.model_name', 'a_simple_model'); +sess_opt.add_session_config_entry('azure.model_version', '1'); # optional, default 1 +sess_opt.add_session_config_entry('azure.verbose', 'true'); # optional, default false + +sess = InferenceSession('a_simple_model.onnx', sess_opt, providers=['CPUExecutionProvider','azureExecutionProvider']) + +run_opt = RunOptions() +run_opt.add_run_config_entry('use_azure', '1') # optional, default '0' to run inference locally. +run_opt.add_run_config_entry('azure.auth_key', '...') # optional, required only when use_azure set to 1 + +x = np.array([1,2,3,4]).astype(np.float32) +y = np.array([4,3,2,1]).astype(np.float32) + +z = sess.run(None, {'X':x, 'Y':y}, run_opt)[0] +``` \ No newline at end of file diff --git a/docs/execution-providers/add-execution-provider.md b/docs/execution-providers/add-execution-provider.md index ad3d260f54..82012820e6 100644 --- a/docs/execution-providers/add-execution-provider.md +++ b/docs/execution-providers/add-execution-provider.md @@ -2,7 +2,7 @@ title: Add a new provider description: Instructions to add a new execution provider to ONNX Runtime parent: Execution Providers -nav_order: 12 +nav_order: 13 redirect_from: /docs/how-to/add-execution-provider --- diff --git a/docs/execution-providers/community-maintained/index.md b/docs/execution-providers/community-maintained/index.md index b62d67e3a0..2eb0670062 100644 --- a/docs/execution-providers/community-maintained/index.md +++ b/docs/execution-providers/community-maintained/index.md @@ -2,7 +2,7 @@ title: Community-maintained parent: Execution Providers has_children: true -nav_order: 11 +nav_order: 12 --- # Community-maintained Providers This list of providers for specialized hardware is contributed and maintained by ONNX Runtime community partners. diff --git a/docs/execution-providers/index.md b/docs/execution-providers/index.md index 34116ca5e6..66d21e3560 100644 --- a/docs/execution-providers/index.md +++ b/docs/execution-providers/index.md @@ -31,7 +31,7 @@ ONNX Runtime supports many different execution providers today. Some of the EPs |Default CPU|[NVIDIA CUDA](../execution-providers/CUDA-ExecutionProvider.md)|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[Rockchip NPU](../execution-providers/community-maintained/RKNPU-ExecutionProvider.md) (*preview*)| |[Intel DNNL](../execution-providers/oneDNN-ExecutionProvider.md)|[NVIDIA TensorRT](../execution-providers/TensorRT-ExecutionProvider.md)|[ARM Compute Library](../execution-providers/community-maintained/ACL-ExecutionProvider.md) (*preview*)|[Xilinx Vitis-AI](../execution-providers/community-maintained/Vitis-AI-ExecutionProvider.md) (*preview*)| |[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)|[DirectML](../execution-providers/DirectML-ExecutionProvider.md)|[Android Neural Networks API](../execution-providers/NNAPI-ExecutionProvider.md)|[Huawei CANN](../execution-providers/community-maintained/CANN-ExecutionProvider.md) (*preview*)| -|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[AMD MIGraphX](../execution-providers/community-maintained/MIGraphX-ExecutionProvider.md) (*preview*)|[ARM-NN](../execution-providers/community-maintained/ArmNN-ExecutionProvider.md) (*preview*)| +|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[AMD MIGraphX](../execution-providers/community-maintained/MIGraphX-ExecutionProvider.md) (*preview*)|[ARM-NN](../execution-providers/community-maintained/ArmNN-ExecutionProvider.md) (*preview*)|[Azure](../execution-providers/Azure-ExecutionProvider.md) (*preview*)| ||[AMD ROCm](../execution-providers/ROCm-ExecutionProvider.md) (*preview*)|[CoreML](../execution-providers/CoreML-ExecutionProvider.md) (*preview*)| ||[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)|[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)| ||[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[Qualcomm SNPE](../execution-providers/SNPE-ExecutionProvider.md)| diff --git a/docs/performance/tune-performance.md b/docs/performance/tune-performance.md index f582ad173c..d0c93ee52e 100644 --- a/docs/performance/tune-performance.md +++ b/docs/performance/tune-performance.md @@ -169,7 +169,70 @@ Currently, there are no special provisions to employ mimalloc on Linux. It is re ### Thread management -* Use the appropriate ORT API to set intra and inter op num threads. Inter op num threads is only used when parallel execution is enabled. +#### Set number of intra-op threads + +Onnxruntime sessions utilize multi-threading to parallelize computation inside each operator. +Customer could configure the number of threads like: + +```python +sess_opt = SessionOptions() +sess_opt.intra_op_num_threads = 3 +sess = ort.InferenceSession('model.onnx', sess_opt) +``` + +With above configuration, two threads will be created in the pool, so along with the main calling thread, there will be three threads in total to participate in intra-op computation. +By default, each session will create one thread per phyical core (except the 1st core) and attach the thread to that core. +However, if customer explicitly set the number of threads like showcased above, there will be no affinity set to any of the created thread. + +In addition, Onnxruntime also allow customers to create a global intra-op thread pool to prevent overheated contentions among session thread pools, please find its usage [here](https://github.com/microsoft/onnxruntime/blob/68b5b2d7d33b6aa2d2b5cf8d89befb4a76e8e7d8/onnxruntime/test/global_thread_pools/test_main.cc#L98). + +#### Set number of inter-op threads + +A inter-op thread pool is for parallelism between operators, and will only be created when session execution mode set to parallel: + +```python +sess_opt = SessionOptions() +sess_opt.execution_mode = ExecutionMode.ORT_PARALLEL +sess_opt.inter_op_num_threads = 3 +sess = ort.InferenceSession('model.onnx', sess_opt) +``` + +By default, inter-op thread pool will also have one thread per physical core. + +#### Set intra-op thread affinity + +For certain scenarios, it may be beneficial to customize intra-op thread affinities, for example: +* There are multiple sessions run in parallel, customer might prefer their intra-op thread pools run on separate cores to avoid contention. +* Customer want to limit a intra-op thread pool to run on only one of the NUMA nodes to reduce overhead of expensive cache miss among nodes. + +For session intra-op thread pool, please read the [configuration](https://github.com/microsoft/onnxruntime/blob/68b5b2d7d33b6aa2d2b5cf8d89befb4a76e8e7d8/include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h#L180) and consume it like: + +```python +sess_opt = SessionOptions() +sess_opt.intra_op_num_threads = 3 +sess_opt.add_session_config_entry('session.intra_op_thread_affinities', '1;2') +sess = ort.InferenceSession('model.onnx', sess_opt, ...) +``` + +For global thread pool, please read the [API](https://github.com/microsoft/onnxruntime/blob/68b5b2d7d33b6aa2d2b5cf8d89befb4a76e8e7d8/include/onnxruntime/core/session/onnxruntime_c_api.h#L3636) and [usage](https://github.com/microsoft/onnxruntime/blob/68b5b2d7d33b6aa2d2b5cf8d89befb4a76e8e7d8/onnxruntime/test/global_thread_pools/test_main.cc#L98). + +#### Numa support and performance tuning + +Since release 1.14, Onnxruntime thread pool could utilize all physical cores that are available over NUMA nodes. +The intra-op thread pool will create a thread on every physical core (except the 1st core). E.g. assume there is a system of 2 NUMA nodes, each has 24 cores. +Hence intra-op thread pool will create 47 threads, and set thread affinity to each core. + +For NUMA systems, it is recommended to test a few thread settings to explore for best performance, in that threads allocated among NUMA nodes might has higher cache-miss overhead when cooperating with each other. For example, when number of intra-op threads has to be 8, there are different ways to set affinity: + +``` +sess_opt = SessionOptions() +sess_opt.intra_op_num_threads = 8 +sess_opt.add_session_config_entry('session.intra_op_thread_affinities', '3,4;5,6;7,8;9,10;11,12;13,14;15,16') # set affinities of all 7 threads to cores in the first NUMA node +# sess_opt.add_session_config_entry('session.intra_op_thread_affinities', '3,4;5,6;7,8;9,10;49,50;51,52;53,54') # set affinities for first 4 threads to the first NUMA node, and others to the second +sess = ort.InferenceSession('resnet50.onnx', sess_opt, ...) +``` + +Test showed that setting affinities to a single NUMA node has nearly 20 percent performance improvement aginst the other case. #### Custom threading callbacks diff --git a/index.html b/index.html index b97957b677..dd714fe6cb 100644 --- a/index.html +++ b/index.html @@ -234,7 +234,9 @@
ACL (Preview)
- ArmNN (Preview)
+ ArmNN (Preview) +
+ Azure (Preview)
CANN (Preview)
@@ -249,6 +251,8 @@ Vitis AI (Preview)
XNNPACK (Preview)
+
+ Azure (Preview)
diff --git a/js/script.js b/js/script.js index d81ffcd28a..9be8a9b99b 100644 --- a/js/script.js +++ b/js/script.js @@ -1223,6 +1223,12 @@ var validCombos = { "linux,C++,X64,CANN": "Follow build instructions from here.", + + "windows,Python,X64,Azure": + "Follow build instructions from here", + + "linux,Python,X64,Azure": + "Follow build instructions from here", }; function commandMessage(key) {