diff --git a/docs/execution-providers/Azure-ExecutionProvider.md b/docs/execution-providers/Azure-ExecutionProvider.md index cb153451a6..5bffb67d0a 100644 --- a/docs/execution-providers/Azure-ExecutionProvider.md +++ b/docs/execution-providers/Azure-ExecutionProvider.md @@ -3,17 +3,25 @@ title: Cloud - Azure description: Instructions to infer an ONNX model remotely with an Azure endpoint parent: Execution Providers nav_order: 11 +redirect_from: /docs/reference/execution-providers/Azure-ExecutionProvider --- # Azure Execution Provider (Preview) {: .no_toc } +The Azure Execution Provider enables ONNX Runtime to invoke a remote Azure endpoint for inference, the endpoint must be deployed or available beforehand. -The Azure Execution Provider enables ONNX Runtime to invoke a remote Azure endpoint for inference. The endpoint must be deployed beforehand. -To consume the endpoint, a model with same inputs and outputs must be first loaded locally. -One use case for Azure Execution Provider is for small-big models. E.g. A smaller model can be deployed on edge devices for faster inference, -while a bigger model can be deployed on Azure for higher precision. Using the Azure Execution Provider, switching between the two can be easily achieved (assuming same inputs and outputs). +Since 1.16, below pluggable operators are available from [onnxruntime-extensions](https://github.com/microsoft/onnxruntime-extensions): + +- [OpenAIAudioToText](https://github.com/microsoft/onnxruntime-extensions/blob/main/docs/custom_ops.md#openaiaudiototext) +- [AzureTextToText](https://github.com/microsoft/onnxruntime-extensions/blob/main/docs/custom_ops.md#azuretexttotext) +- [AzureTritonInvoker](https://github.com/microsoft/onnxruntime-extensions/blob/main/docs/custom_ops.md#azuretritoninvoker) + +With the operators, Azure Execution Provider supports two mode of usage: + +- [Edge and azure side by side](#Edge-and-azure-side-by-side) +- [Merge and run the hybrid](#Merge-and-run-the-hybrid) Azure Execution Provider is in preview stage, and all API(s) and usage are subject to change. @@ -24,11 +32,10 @@ Azure Execution Provider is in preview stage, and all API(s) and usage are subje {:toc} ## Install -Pre-built Python binaries of ONNX Runtime with Azure EP are published on Pypi: [onnxruntime-azure](https://pypi.org/project/onnxruntime-azure/) +Since 1.16, Azure Execution Provider is shipped by default in both python and nuget packages. ## Requirements - -For Linux, please make sure openssl is installed. +Since 1.16, all Azure Execution Provider operators are shipped with [onnxruntime-extensions](https://github.com/microsoft/onnxruntime-extensions) (>=v0.9.0) python and nuget packages. Please ensure the installation of correct onnxruntime-extension packages before using Azure Execution Provider. ## Build @@ -36,36 +43,124 @@ For build instructions, please see the [BUILD page](../build/eps.md#azure). ## Usage -### Python - +### Edge and azure side by side +In this mode, there are two models running simultaneously. The azure model runs asynchronously by [RunAsync](https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/session/onnxruntime_c_api.h#L4341) API, which is also available through [python](https://github.com/microsoft/onnxruntime/blob/873ef8b8f0b09b49c0a7b7e2f03f3639d7418c22/onnxruntime/python/onnxruntime_pybind_state.cc#L1759) and [csharp](https://github.com/microsoft/onnxruntime/blob/873ef8b8f0b09b49c0a7b7e2f03f3639d7418c22/csharp/src/Microsoft.ML.OnnxRuntime/InferenceSession.shared.cs#L1147). ```python -from onnxruntime import * -import numpy as np import os +import onnx +from onnx import helper, TensorProto +from onnxruntime_extensions import get_library_path +from onnxruntime import SessionOptions, InferenceSession +import numpy as np +import threading -sess_opt = SessionOptions() -sess_opt.add_session_config_entry('azure.endpoint_type', 'triton'); # only support triton server for now -sess_opt.add_session_config_entry('azure.uri', 'https://...') -sess_opt.add_session_config_entry('azure.model_name', 'a_simple_model'); -sess_opt.add_session_config_entry('azure.model_version', '1'); # optional, default 1 -sess_opt.add_session_config_entry('azure.verbose', 'true'); # optional, default false -sess = InferenceSession('a_simple_model.onnx', sess_opt, providers=['CPUExecutionProvider','azureExecutionProvider']) +# Generate the local model by: +# https://github.com/microsoft/onnxruntime-extensions/blob/main/tutorials/whisper_e2e.py +def get_whiper_tiny(): + return '/onnxruntime-extensions/tutorials/whisper_onnx_tiny_en_fp32_e2e.onnx' -run_opt = RunOptions() -run_opt.add_run_config_entry('use_azure', '1') # optional, default '0' to run inference locally. -run_opt.add_run_config_entry('azure.auth_key', '...') # optional, required only when use_azure set to 1 -x = np.array([1,2,3,4]).astype(np.float32) -y = np.array([4,3,2,1]).astype(np.float32) +# Generate the azure model +def get_openai_audio_azure_model(): + auth_token = helper.make_tensor_value_info('auth_token', TensorProto.STRING, [1]) + model = helper.make_tensor_value_info('model_name', TensorProto.STRING, [1]) + response_format = helper.make_tensor_value_info('response_format', TensorProto.STRING, [-1]) + file = helper.make_tensor_value_info('file', TensorProto.UINT8, [-1]) -z = sess.run(None, {'X':x, 'Y':y}, run_opt)[0] + transcriptions = helper.make_tensor_value_info('transcriptions', TensorProto.STRING, [-1]) + + invoker = helper.make_node('OpenAIAudioToText', + ['auth_token', 'model_name', 'response_format', 'file'], + ['transcriptions'], + domain='com.microsoft.extensions', + name='audio_invoker', + model_uri='https://api.openai.com/v1/audio/transcriptions', + audio_format='wav', + verbose=False) + + graph = helper.make_graph([invoker], 'graph', [auth_token, model, response_format, file], [transcriptions]) + model = helper.make_model(graph, ir_version=8, + opset_imports=[helper.make_operatorsetid('com.microsoft.extensions', 1)]) + model_name = 'openai_whisper_azure.onnx' + onnx.save(model, model_name) + return model_name + + +if __name__ == '__main__': + sess_opt = SessionOptions() + sess_opt.register_custom_ops_library(get_library_path()) + + azure_model_path = get_openai_audio_azure_model() + azure_model_sess = InferenceSession(azure_model_path, + sess_opt, providers=['CPUExecutionProvider', 'AzureExecutionProvider']) # load AzureEP + + with open('test16.wav', "rb") as _f: # read raw audio data from a local wav file + audio_stream = np.asarray(list(_f.read()), dtype=np.uint8) + + azure_model_inputs = { + "auth_token": np.array([os.getenv('AUDIO', '')]), # read auth from env variable + "model_name": np.array(['whisper-1']), + "response_format": np.array(['text']), + "file": audio_stream + } + + + class RunAsyncState: + def __init__(self): + self.__event = threading.Event() + self.__outputs = None + self.__err = '' + + def fill_outputs(self, outputs, err): + self.__outputs = outputs + self.__err = err + self.__event.set() + + def get_outputs(self): + if self.__err != '': + raise Exception(self.__err) + return self.__outputs; + + def wait(self, sec): + self.__event.wait(sec) + + + def azureRunCallback(outputs: np.ndarray, state: RunAsyncState, err: str) -> None: + state.fill_outputs(outputs, err) + + + run_async_state = RunAsyncState(); + # infer azure model asynchronously + azure_model_sess.run_async(None, azure_model_inputs, azureRunCallback, run_async_state) + + # in the same time, run the edge + edge_model_path = get_whiper_tiny() + edge_model_sess = InferenceSession(edge_model_path, + sess_opt, providers=['CPUExecutionProvider']) + + edge_model_outputs = edge_model_sess.run(None, { + 'audio_stream': np.expand_dims(audio_stream, 0), + 'max_length': np.asarray([200], dtype=np.int32), + 'min_length': np.asarray([0], dtype=np.int32), + 'num_beams': np.asarray([2], dtype=np.int32), + 'num_return_sequences': np.asarray([1], dtype=np.int32), + 'length_penalty': np.asarray([1.0], dtype=np.float32), + 'repetition_penalty': np.asarray([1.0], dtype=np.float32) + }) + + print("\noutput from whisper tiny: ", edge_model_outputs) + run_async_state.wait(10) + print("\nresponse from openAI: ", run_async_state.get_outputs()) + # compare results and pick the better ``` -### Current Limitations +### Merge and run the hybrid -* Only supports [Triton Inference Server](https://github.com/triton-inference-server) on [AML](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-with-triton?tabs=python%2Cendpoint). -* Only builds and run on Windows and Linux. -* Available only as Python package, but can be built from source and used via C/C++ API(s). -* **Known Issue:** For certain ubuntu versions, https call made by AzureEP might report error - "error setting certificate verify location ...". -To silence it, please create file "/etc/pki/tls/certs/ca-bundles.crt" that link to "/etc/ssl/certs/ca-certificates.crt". \ No newline at end of file +Alternatively, one could also merge local and azure models into a hybrid, then infer as an ordinary onnx model. +Sample scripts could be found [here](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/python/AzureEP). + +## Current Limitations + +* Only builds and run on Windows, Linux and Android. +* For Android, AzureTritonInvoker is not supported. \ No newline at end of file diff --git a/docs/execution-providers/index.md b/docs/execution-providers/index.md index f2a1fe854f..1e2c13abcf 100644 --- a/docs/execution-providers/index.md +++ b/docs/execution-providers/index.md @@ -26,7 +26,7 @@ ONNX Runtime supports many different execution providers today. Some of the EPs |Default CPU|[NVIDIA CUDA](../execution-providers/CUDA-ExecutionProvider.md)|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[Rockchip NPU](../execution-providers/community-maintained/RKNPU-ExecutionProvider.md) (*preview*)| |[Intel DNNL](../execution-providers/oneDNN-ExecutionProvider.md)|[NVIDIA TensorRT](../execution-providers/TensorRT-ExecutionProvider.md)|[ARM Compute Library](../execution-providers/community-maintained/ACL-ExecutionProvider.md) (*preview*)|[Xilinx Vitis-AI](../execution-providers/Vitis-AI-ExecutionProvider.md) (*preview*)| |[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)|[DirectML](../execution-providers/DirectML-ExecutionProvider.md)|[Android Neural Networks API](../execution-providers/NNAPI-ExecutionProvider.md)|[Huawei CANN](../execution-providers/community-maintained/CANN-ExecutionProvider.md) (*preview*)| -|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[AMD MIGraphX](../execution-providers/MIGraphX-ExecutionProvider.md)|[ARM-NN](../execution-providers/community-maintained/ArmNN-ExecutionProvider.md) (*preview*)| +|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[AMD MIGraphX](../execution-providers/MIGraphX-ExecutionProvider.md)|[ARM-NN](../execution-providers/community-maintained/ArmNN-ExecutionProvider.md) (*preview*)|[AZURE](../execution-providers/Azure-ExecutionProvider.md) (*preview*)| |[XNNPACK](../execution-providers/Xnnpack-ExecutionProvider.md)|[Intel OpenVINO](../execution-providers/OpenVINO-ExecutionProvider.md)|[CoreML](../execution-providers/CoreML-ExecutionProvider.md) (*preview*)| ||[AMD ROCm](../execution-providers/ROCm-ExecutionProvider.md)|[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)| ||[TVM](../execution-providers/community-maintained/TVM-ExecutionProvider.md) (*preview*)|[Qualcomm QNN](../execution-providers/QNN-ExecutionProvider.md)|