mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-13 18:08:13 +00:00
Lots of minor doc improvements (#9930)
This commit is contained in:
Gary Miguel
GitHub
parent
2f3511540c
commit
41a714bbb3
5 changed files with 73 additions and 80 deletions
|
|
@ -1,5 +1,5 @@
|
|||
---
|
||||
title: Direct ML
|
||||
title: DirectML
|
||||
description: Instructions to execute ONNX Runtime with the DirectML execution provider
|
||||
parent: Execution Providers
|
||||
nav_order: 5
|
||||
|
|
@ -9,7 +9,7 @@ redirect_from: /docs/reference/execution-providers/DirectML-ExecutionProvider
|
|||
# DirectML Execution Provider
|
||||
{: .no_toc }
|
||||
|
||||
The DirectML Execution Provider is a component of ONNX Runtime that uses DirectML to accelerate inference of ONNX models. The DirectML execution provider is capable of greatly improving evaluation time of models using commodity GPU hardware, without sacrificing broad hardware support or requiring vendor-specific extensions to be installed.
|
||||
The DirectML Execution Provider is a component of ONNX Runtime that uses [DirectML](https://docs.microsoft.com/en-us/windows/ai/directml/dml-intro) to accelerate inference of ONNX models. The DirectML execution provider is capable of greatly improving evaluation time of models using commodity GPU hardware, without sacrificing broad hardware support or requiring vendor-specific extensions to be installed.
|
||||
|
||||
DirectML is a high-performance, hardware-accelerated DirectX 12 library for machine learning on Windows. DirectML provides GPU acceleration for common machine learning tasks across a broad range of supported hardware and drivers.
|
||||
|
||||
|
|
@ -39,7 +39,7 @@ DirectML is compatible with Windows 10, version 1709 (10.0.16299; RS3, "Fall Cre
|
|||
## Build
|
||||
|
||||
Requirements for building the DirectML execution provider:
|
||||
1. Visual Studio 2017 toolchain
|
||||
1. Visual Studio 2017 toolchain
|
||||
2. [The Windows 10 SDK (10.0.18362.0) for Windows 10, version 1903](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk) (or newer)
|
||||
|
||||
To build onnxruntime with the DML EP included, supply the `--use_dml` parameter to `build.bat`. e.g.
|
||||
|
|
@ -126,6 +126,6 @@ The DirectML execution provider currently supports ONNX opset 12 ([ONNX v1.7](ht
|
|||
## Additional Resources
|
||||
|
||||
* [DirectML documentation \(docs.microsoft.com\)](https://docs.microsoft.com/en-us/windows/win32/direct3d12/dml)
|
||||
* [DMLCreateDevice function](https://docs.microsoft.com/windows/win32/api/directml/nf-directml-dmlcreatedevice)
|
||||
* [ID3D12Device::CreateCommandQueue method](https://docs.microsoft.com/windows/win32/api/d3d12/nf-d3d12-id3d12device-createcommandqueue)
|
||||
* [DMLCreateDevice function](https://docs.microsoft.com/windows/win32/api/directml/nf-directml-dmlcreatedevice)
|
||||
* [ID3D12Device::CreateCommandQueue method](https://docs.microsoft.com/windows/win32/api/d3d12/nf-d3d12-id3d12device-createcommandqueue)
|
||||
* [Direct3D 12 programming guide](https://docs.microsoft.com/windows/win32/direct3d12/directx-12-programming-guide)
|
||||
|
|
|
|||
|
|
@ -14,7 +14,7 @@ This allows scenarios such as passing a [Windows.Media.VideoFrame](https://docs.
|
|||
|
||||
The WinML API is a WinRT API that shipped inside the Windows OS starting with build 1809 (RS5) in the Windows.AI.MachineLearning namespace. It embedded a version of the ONNX Runtime.
|
||||
|
||||
In addition to using the in-box version of WinML, WinML can also be installed as an application redistributable package (see [layered architecture](../reference/high-level-design.md#the-onnx-runtime-and-windows-os-integration) for technical details).
|
||||
In addition to using the in-box version of WinML, WinML can also be installed as an application redistributable package (see [layered architecture](../reference/high-level-design#windows-os-integration) for technical details).
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
|
@ -39,7 +39,7 @@ Windows 8.1+
|
|||
|
||||
Any code already written for the Windows.AI.MachineLearning API can be easily modified to run against the Microsoft.ML.OnnxRuntime package. All types originally referenced by inbox customers via the Windows namespace will need to be updated to now use the Microsoft namespace.
|
||||
|
||||
* [Samples in Github](https://github.com/microsoft/Windows-Machine-Learning/tree/master/Samples/SqueezeNetObjectDetection/Desktop/cpp)
|
||||
* [Samples in Github](https://github.com/microsoft/Windows-Machine-Learning/tree/master/Samples/SqueezeNetObjectDetection/Desktop/cpp)
|
||||
|
||||
## Should I use the in-box vs NuGet WinML version?
|
||||
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ redirect_from: /docs/how-to/install
|
|||
# Install ONNX Runtime (ORT)
|
||||
|
||||
|
||||
See the [installation matrix](https://onnxruntime.ai) for recommended instructions for desired combinations of target operating system, hardware, accelerator, and language.
|
||||
See the [installation matrix](https://onnxruntime.ai) for recommended instructions for desired combinations of target operating system, hardware, accelerator, and language.
|
||||
|
||||
Details on OS versions, compilers, language versions, dependent libraries, etc can be found under [Compatibility](../reference/compatibility).
|
||||
|
||||
|
|
@ -53,21 +53,21 @@ pip install skl2onnx
|
|||
### Install ONNX Runtime (ORT)
|
||||
|
||||
```bash
|
||||
# CPU
|
||||
dotnet add package Microsoft.ML.OnnxRuntime --version 1.8.1
|
||||
# CPU
|
||||
dotnet add package Microsoft.ML.OnnxRuntime
|
||||
```
|
||||
```bash
|
||||
# GPU
|
||||
dotnet add package Microsoft.ML.OnnxRuntime.Gpu --version 1.8.1
|
||||
dotnet add package Microsoft.ML.OnnxRuntime.Gpu
|
||||
```
|
||||
```bash
|
||||
# DirectML
|
||||
dotnet add package Microsoft.ML.OnnxRuntime.DirectML --version 1.8.1
|
||||
dotnet add package Microsoft.ML.OnnxRuntime.DirectML
|
||||
```
|
||||
|
||||
```bash
|
||||
# WinML
|
||||
dotnet add package Microsoft.AI.MachineLearning --version 1.8.1
|
||||
dotnet add package Microsoft.AI.MachineLearning
|
||||
```
|
||||
|
||||
## Install on web and mobile
|
||||
|
|
@ -214,7 +214,7 @@ by running `locale-gen en_US.UTF-8` and `update-locale LANG=en_US.UTF-8`
|
|||
|
||||
## Training install table for all languages
|
||||
|
||||
ONNX Runtime Training packages are available for different versions of PyTorch, CUDA and ROCm versions.
|
||||
ONNX Runtime Training packages are available for different versions of PyTorch, CUDA and ROCm versions.
|
||||
|
||||
The install command is:
|
||||
```cmd
|
||||
|
|
@ -231,4 +231,4 @@ The _location_ needs to be specified for any specific version other than the def
|
|||
|PyTorch 1.9 (CUDA 10.2) **Default**|[**onnxruntime-training**](https://pypi.org/project/onnxruntime-training/)|[onnxruntime_nightly_torch190.cu102](https://onnxruntimepackages.z14.web.core.windows.net/onnxruntime_nightly_torch190.cu102.html)|
|
||||
|PyTorch 1.9 (CUDA 11.1)|[**onnxruntime_stable_torch190.cu111**](https://onnxruntimepackages.z14.web.core.windows.net/onnxruntime_stable_torch190.cu111.html)|[onnxruntime_nightly_torch190.cu111](https://onnxruntimepackages.z14.web.core.windows.net/onnxruntime_nightly_torch190.cu111.html)|
|
||||
|[*Preview*] PyTorch 1.8.1 (ROCm 4.2)|[**onnxruntime_stable_torch181.rocm42**](https://onnxruntimepackages.z14.web.core.windows.net/onnxruntime_stable_torch181.rocm42.html)|[onnxruntime_nightly_torch181.rocm42](https://onnxruntimepackages.z14.web.core.windows.net/onnxruntime_nightly_torch181.rocm42.html)|
|
||||
|[*Preview*] PyTorch 1.9 (ROCm 4.2)|[**onnxruntime_stable_torch190.rocm42**](https://onnxruntimepackages.z14.web.core.windows.net/onnxruntime_stable_torch190.rocm42.html)|[onnxruntime_nightly_torch190.rocm42](https://onnxruntimepackages.z14.web.core.windows.net/onnxruntime_nightly_torch190.rocm42.html)|
|
||||
|[*Preview*] PyTorch 1.9 (ROCm 4.2)|[**onnxruntime_stable_torch190.rocm42**](https://onnxruntimepackages.z14.web.core.windows.net/onnxruntime_stable_torch190.rocm42.html)|[onnxruntime_nightly_torch190.rocm42](https://onnxruntimepackages.z14.web.core.windows.net/onnxruntime_nightly_torch190.rocm42.html)|
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ redirect_from: /docs/how-to/graph-optimizations
|
|||
# Graph Optimizations in ONNX Runtime
|
||||
{: .no_toc }
|
||||
|
||||
ONNX Runtime provides various graph optimizations to improve model performance. Graph optimizations are essentially graph-level transformations, ranging from small graph simplifications and node eliminations to more complex node fusions and layout optimizations.
|
||||
ONNX Runtime provides various graph optimizations to improve performance. Graph optimizations are essentially graph-level transformations, ranging from small graph simplifications and node eliminations to more complex node fusions and layout optimizations.
|
||||
|
||||
Graph optimizations are divided in several categories (or *levels*) based on their complexity and functionality. They can be performed either *online* or *offline*. In online mode, the optimizations are done before performing the inference, while in offline mode, the runtime saves the optimized graph to disk. ONNX Runtime provides Python, C#, C++, and C APIs to enable different optimization levels and to choose between offline vs. online mode.
|
||||
|
||||
|
|
@ -22,10 +22,11 @@ Below we provide details on the optimization levels, the online/offline mode, an
|
|||
|
||||
## Graph Optimization Levels
|
||||
|
||||
Graph optimizations are divided in three levels:
|
||||
* Basic
|
||||
* Extended
|
||||
* Layout Optimizations
|
||||
Graph optimizations are divided into three levels:
|
||||
|
||||
1. Basic
|
||||
2. Extended
|
||||
3. Layout Optimizations
|
||||
|
||||
The optimizations belonging to one level are performed after the optimizations of the previous level have been applied (e.g., extended optimizations are applied after basic optimizations have been applied).
|
||||
|
||||
|
|
@ -56,20 +57,26 @@ These optimizations include complex node fusions. They are run after graph parti
|
|||
|
||||
| Optimization | Execution Provider | Comment |
|
||||
|---------------------------------|--------------------|-----------------------------------------------------------------------------|
|
||||
| GEMM Activation Fusion | cpu | |
|
||||
| Matmul Add Fusion | cpu | |
|
||||
| Conv Activation Fusion | cpu | |
|
||||
| GELU Fusion | cpu or cuda | |
|
||||
| Layer Normalization Fusion | cpu or cuda | |
|
||||
| BERT Embedding Layer Fusion | cpu or cuda | Fuse BERT embedding layer, layer normalization and attention mask length |
|
||||
| Attention Fusion | cpu or cuda | Attention mask has approximation in cuda execution provider |
|
||||
| Skip Layer Normalization Fusion | cpu or cuda | Fuse bias of fully connected layer, skip connection and layer normalization |
|
||||
| Bias GELU Fusion | cpu or cuda | Fuse bias of fully connected layer and GELU activation |
|
||||
| GELU Approximation | cuda | Erf is approximated by a formula using tanh function |
|
||||
| GEMM Activation Fusion | CPU | |
|
||||
| Matmul Add Fusion | CPU | |
|
||||
| Conv Activation Fusion | CPU | |
|
||||
| GELU Fusion | CPU or CUDA | |
|
||||
| Layer Normalization Fusion | CPU or CUDA | |
|
||||
| BERT Embedding Layer Fusion | CPU or CUDA | Fuse BERT embedding layer, layer normalization and attention mask length |
|
||||
| Attention Fusion* | CPU or CUDA | |
|
||||
| Skip Layer Normalization Fusion | CPU or CUDA | Fuse bias of fully connected layer, skip connection and layer normalization |
|
||||
| Bias GELU Fusion | CPU or CUDA | Fuse bias of fully connected layer and GELU activation |
|
||||
| GELU Approximation* | CUDA | Disabled by default. Enable with [kOrtSessionOptionsEnableGeluApproximation](https://cs.github.com/microsoft/onnxruntime/blob/175acf08f470db0bb2e4b8eefe55cdeb87c8b132/include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h?q=kOrtSessionOptionsEnableGeluApproximation#L52) |
|
||||
|
||||
To optimize inference performance of BERT model, approximation is used in GELU approximation and Attention fusion for cuda execution provider. There might be slight difference in result. The impact on accuracy could be neglected based on our evaluation: F1 score for a BERT model on SQuAD v1.1 is almost same (87.05 vs 87.03).
|
||||
|
||||
GELU approximation is disabled by default.
|
||||
<details>
|
||||
<summary>
|
||||
Approximations (click to expand)
|
||||
</summary>
|
||||
|
||||
To optimize performance of [BERT](https://en.wikipedia.org/wiki/BERT_(language_model)), approximation is used in GELU Approximation and Attention Fusion for CUDA execution provider. The impact on accuracy is negligible based on our evaluation: F1 score for a BERT model on SQuAD v1.1 is almost same (87.05 vs 87.03).
|
||||
|
||||
</details>
|
||||
|
||||
### Layout Optimizations
|
||||
|
||||
|
|
@ -79,17 +86,17 @@ These optimizations change the data layout for applicable nodes to achieve highe
|
|||
|
||||
## Online/Offline Mode
|
||||
|
||||
All optimizations can be performed either online or offline. In online mode, when initializing an inference session, we also apply all enabled graph optimizations before performing model inference. Applying all optimizations each time we initiate a session can add overhead to the model startup time (especially for complex models), which can be critical in production scenarios. This is where the offline mode can bring a lot of benefit. In offline mode, after performing graph optimizations, ONNX Runtime serializes the resulting model to disk. Subsequently, when new inference sessions are created for this model, we can instead use the already optimized model to reduce startup time.
|
||||
All optimizations can be performed either online or offline. In online mode, when initializing an inference session, we also apply all enabled graph optimizations before performing model inference. Applying all optimizations each time we initiate a session can add overhead to the model startup time (especially for complex models), which can be critical in production scenarios. This is where the offline mode can bring a lot of benefit. In offline mode, after performing graph optimizations, ONNX Runtime serializes the resulting model to disk. Subsequently, we can reduce startup time by using the already optimized model and disabling all optimizations.
|
||||
|
||||
**Notes**:
|
||||
**Notes**:
|
||||
|
||||
* When running in offline mode, make sure to use the exact same options (e.g., execution providers, optimization level) and hardware as the target machine that the model inference will run on (e.g., you cannot run a model pre-optimized for a GPU execution provider on a machine that is equipped only with CPU).
|
||||
* When layout optimizations are enabled, the offline mode can only be used on compatible hardware to the environment when the offline model is saved. For example, if model has layout optimized for AVX2, the offline model would require CPUs that support AVX2.
|
||||
|
||||
## Usage
|
||||
|
||||
### General Note
|
||||
**Levels**:
|
||||
### Levels
|
||||
|
||||
ONNX Runtime defines the `GraphOptimizationLevel` enum to determine which of the aforementioned optimization levels will be enabled. Choosing a level enables the optimizations of that level, as well as the optimizations of all preceding levels. For example, enabling Extended optimizations, also enables Basic optimizations. The mapping of these levels to the enum is as follows:
|
||||
|
||||
* GraphOptimizationLevel::ORT_DISABLE_ALL -> Disables all optimizations
|
||||
|
|
@ -97,10 +104,11 @@ ONNX Runtime defines the `GraphOptimizationLevel` enum to determine which of the
|
|||
* GraphOptimizationLevel::ORT_ENABLE_EXTENDED -> Enables basic and extended optimizations
|
||||
* GraphOptimizationLevel::ORT_ENABLE_ALL -> Enables all available optimizations including layout optimizations
|
||||
|
||||
**Online/Offline Mode**:
|
||||
To enable serialization of the optimized model to disk, set the SessionOptions option `optimized_model_path` to the desired path where the optimized model will be stored.
|
||||
### Offline mode
|
||||
|
||||
### Python API Example
|
||||
To enable serialization of the optimized model to disk, set the SessionOptions option `optimized_model_filepath`.
|
||||
|
||||
#### Python API Example
|
||||
```python
|
||||
import onnxruntime as rt
|
||||
|
||||
|
|
@ -115,7 +123,7 @@ sess_options.optimized_model_filepath = "<model_output_path\optimized_model.onnx
|
|||
session = rt.InferenceSession("<model_path>", sess_options)
|
||||
```
|
||||
|
||||
### C API Example
|
||||
#### C API Example
|
||||
```c
|
||||
const OrtApi* Ort::g_api = OrtGetApi(ORT_API_VERSION);
|
||||
OrtEnv* env;
|
||||
|
|
@ -135,7 +143,7 @@ session = rt.InferenceSession("<model_path>", sess_options)
|
|||
g_ort->CreateSession(env, model_path, session_option, &session);
|
||||
```
|
||||
|
||||
### C# API Example
|
||||
#### C# API Example
|
||||
```c#
|
||||
SessionOptions so = new SessionOptions();
|
||||
|
||||
|
|
@ -148,7 +156,7 @@ so.OptimizedModelFilePath = "model_output_path\optimized_model.onnx"
|
|||
var session = new InferenceSession(modelPath, so);
|
||||
```
|
||||
|
||||
### C++ API Example
|
||||
#### C++ API Example
|
||||
```c++
|
||||
Ort::SessionOptions session_options;
|
||||
|
||||
|
|
|
|||
|
|
@ -22,11 +22,11 @@ available on disparate platforms.
|
|||
* Provide the right abstraction and runtime support for custom accelerators and
|
||||
runtimes. We call this abstraction an [execution
|
||||
provider](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime/core/framework/execution_provider.h). It defines and exposes a set of
|
||||
its capabilities to ONNXRuntime: a set of single or fused nodes it can
|
||||
its capabilities to ONNX Runtime: a set of single or fused nodes it can
|
||||
execute, its memory allocator, and more. Custom accelerators and runtimes are
|
||||
instances of execution providers.
|
||||
* We don't expect that an execution provider can always run an ONNX model fully
|
||||
on its device. This means that ONNXRuntime must be able to execute a single
|
||||
on its device. This means that ONNX Runtime must be able to execute a single
|
||||
model in a heterogeneous environment involving multiple execution providers.
|
||||
* Provide support for high-level optimizations that can be expressed as
|
||||
model-to-model transformations via a [graph-transformation
|
||||
|
|
@ -38,35 +38,36 @@ rules](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime
|
|||
|
||||
## High-level system architecture
|
||||
|
||||
The flow is quite simple.
|
||||
The flow is quite simple.
|
||||
1. Starting from an ONNX model, ONNX Runtime first
|
||||
converts the model graph into its in-memory graph representation.
|
||||
2. It then applies a number of graph transformations that a) perform a set of provider independent optimizations such cast transformations between float16 and float32, and b) partition the graph into a set of subgraphs based on the available execution providers.
|
||||
3. Each subgraph is assigned to an execution provider. We ensure that a subgraph can be executed by an execution provider by querying the capability of the execution provider using the GetCapability() API.
|
||||
converts the model graph into its in-memory graph representation.
|
||||
2. It performs a set of provider independent [optimizations](../performance/graph-optimizations).
|
||||
3. It partitions the graph into a set of subgraphs based on the available execution providers.
|
||||
4. Each subgraph is assigned to an execution provider. We ensure that a subgraph can be executed by an execution provider by querying the capability of the execution provider using the `GetCapability()` API.
|
||||
|
||||
|
||||
|
||||
|
||||
### More about partitioning
|
||||
{: .no_toc }
|
||||
ONNX Runtime partitions a model graph into subgraphs based on the available execution providers, one for each distinct provider. ONNXRuntime provides a default execution provider that is used as the fallback execution for the
|
||||
ONNX Runtime partitions a model graph into subgraphs based on the available execution providers, one for each distinct provider. ONNX Runtime provides a default execution provider that is used as the fallback execution for the
|
||||
operators that cannot be pushed onto the more specialized but more efficient execution providers. Intuitively we want to push computation to more specialized execution providers whenever possible.
|
||||
|
||||
We use a simple graph partitioning technique. The available execution providers will be considered in a specific order, and each will be assigned the maximal subgraphs (possibly more than one) that it is able to handle. The ONNXRuntime-provided default execution provider will be the last one considered, and it ensures completeness. More sophisticated optimizations can be considered in the future (or can even be implemented as a composite execution provider).
|
||||
We use a simple graph partitioning technique. The available execution providers will be considered in a specific order, and each will be assigned the maximal subgraphs (possibly more than one) that it is able to handle. The ONNX Runtime-provided default execution provider will be the last one considered, and it ensures completeness. More sophisticated optimizations can be considered in the future (or can even be implemented as a composite execution provider).
|
||||
|
||||
Conceptually, each partition is reduced to a single fused operator. It is created by invoking the execution provider's Compile() method and wraps it as a custom operator. Currently we support only synchronous mode of execution. An execution provider exposes its memory allocator, which is used to allocate the input tensors for the execution provider. The rewriting and partitioning transform the initial model graph into a new graph composed of operators assigned to either the default execution provider or other registered execution providers. The ONNXRuntime execution engine is responsible for running this graph.
|
||||
Conceptually, each partition is reduced to a single fused operator. It is created by invoking the execution provider's Compile() method and wraps it as a custom operator. Currently we support only synchronous mode of execution. An execution provider exposes its memory allocator, which is used to allocate the input tensors for the execution provider. The rewriting and partitioning transform the initial model graph into a new graph composed of operators assigned to either the default execution provider or other registered execution providers. The ONNX Runtime execution engine is responsible for running this graph.
|
||||
|
||||
## Key design decisions
|
||||
|
||||
* Multiple threads can invoke the Run() method on the same
|
||||
* Multiple threads can invoke the `Run()` method on the same
|
||||
inference session object. See [API doc](../get-started/with-c.md) for more details.
|
||||
* To facilitate this, the Compute() function of all kernels is const
|
||||
* To facilitate this, the `Compute()` function of all kernels is const
|
||||
implying the kernels are stateless.
|
||||
* Implementations of the operators by execution providers are called
|
||||
kernels. Each execution provider supports a subset of the (ONNX)
|
||||
operators/kernels.
|
||||
* The ONNX Runtime guarantees that all operators are supported by the default
|
||||
* ONNX Runtime guarantees that all operators are supported by the default
|
||||
execution provider.
|
||||
* Tensor representation: ONNX Runtime will utilize a standard representation for
|
||||
* Tensor representation: ONNX Runtime uses a standard representation for
|
||||
the tensor runtime values. The execution providers can internally use a
|
||||
different representation if they choose to, but it is their responsibility to convert the values from/to the standard representation at the boundaries of their subgraph.
|
||||
|
||||
|
|
@ -77,22 +78,18 @@ different representation if they choose to, but it is their responsibility to co
|
|||
* [Add a new graph transform](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime/core/optimizer/graph_transformer.h)
|
||||
* [Add a new rewrite rule](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime/core/optimizer/rewrite_rule.h)
|
||||
|
||||
## The ONNX Runtime and Windows OS integration
|
||||
## Windows OS integration
|
||||
|
||||
ONNX Runtime is available in the Windows operating system starting with build 1809 (RS5)+. The runtime is embedded inside the Windows.AI.MachineLearning.dll and was exposed via the WinRT API (WinML for short). It includes CPU support and a DirectML execution provider for GPU support.
|
||||
ONNX Runtime is available in Windows 10 versions >= 1809 and all versions of Windows 11. It is embedded inside Windows.AI.MachineLearning.dll and exposed via the WinRT API (WinML for short). It includes the CPU execution provider and the [DirectML execution provider](../execution-providers/DirectML-ExecutionProvider) for GPU support.
|
||||
|
||||
The udpated layered architecture to ONNX Runtime and Windows ML is available since ONNX Runtime 1.2.
|
||||
|
||||
The high level design looks like this
|
||||
The high level design looks like this:
|
||||
|
||||
|
||||
|
||||
We replaced the embedded ONNX Runtime with the new ONNXRuntime.dll. With this new approach customers have flexibility on which API they choose to use and on how they want to distribute the binaries.
|
||||
|
||||
### API choice
|
||||
{: .no_toc }
|
||||
|
||||
Developers can now choose which API works best for their scenario.
|
||||
You can choose to use either the WinRT API or the C API.
|
||||
|
||||
||WinRT|C API|
|
||||
|--|--|--|
|
||||
|
|
@ -103,30 +100,18 @@ Developers can now choose which API works best for their scenario.
|
|||
### Distribution choice
|
||||
{: .no_toc }
|
||||
|
||||
You can also choose to use runtimes included in the Windows OS, or use the redist nuget to ship the runtime with the app.
|
||||
You can choose to use the runtime included in Windows, or use the [NuGet package](https://docs.microsoft.com/en-us/nuget/what-is-nuget) to ship the runtime with the app.
|
||||
|
||||
|Distribution|Inbox|App NuGet|
|
||||
|--|--|--|
|
||||
|Disk footprint| Included in the OS| Included in the App|
|
||||
|Servicing fixes| Serviced by OS updates| Serviced by the App|
|
||||
|Execution Providers| CPU & DirectML EP | App chosen EP|
|
||||
|Compatability testing| Tested with OS flights against supported GPUs and CPU's | App performs compatibility testing|
|
||||
|Compatibility testing| Tested with OS flights against supported GPUs and CPU's | App performs compatibility testing|
|
||||
|Opset| Refreshed in OS updates| App chooses|
|
||||
|
||||
### Using the NuGet WinRT API with other C-API distributions
|
||||
{: .no_toc }
|
||||
|
||||
The WinRT API NuGet is distributed with a curated build of the OnnxRuntime engine. App developers may wish to use the WinRT API, but find themselves limited to the functionality provided by the curated OnnxRuntime engine distributed as part of the WinRT API NuGet package. This can happen because the OnnxRuntime engine shipped with the WinRT API NuGet package only contains the CPU and DML execution providers.
|
||||
|
||||
App developers may additionally wish to use a custom build-from-source version of the OnnxRuntime engine as well, or use a prebuilt version of the OnnxRuntime engine from another distribution source like the Micorosoft.ML.OnnxRuntime.MKLML distribution.
|
||||
|
||||
To enable this, the WinRT API NuGet has been made to be compatible with a set of OnnxRuntime engines that ship in different NuGet packages.
|
||||
|
||||
Please refer to the following table listing the distributions with compatible OnnxRuntime engines.
|
||||
|
||||
* [Microsoft.ML.OnnxRuntime](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime)
|
||||
* [Microsoft.ML.OnnxRuntime.DirectML](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.DirectML/)
|
||||
|
||||
Note that compatible distributions must match in release version.
|
||||
|
||||
In order to use compatible engines, replace the onnxruntime.dll with the desired engine binary and its associated binaries.
|
||||
The WinRT API NuGet package is distributed with a specific version of ONNX Runtime, but apps can include their own version of ONNX Runtime (either a [released version](../install/#cccwinml-installs) or [a custom build](../build/)). You may wish to do this to use non-default execution providers.
|
||||
To use your own version of ONNX Runtime, replace onnxruntime.dll with your desired version.
|
||||
|
|
|
|||
Loading…
Reference in a new issue