mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-30 20:18:08 +00:00
Refactor DirectML and Windows docs (#12096)
This commit is contained in:
Nat Kershaw (MSFT)
GitHub
parent
33d47609c4
commit
8c23c57a9f
4 changed files with 71 additions and 59 deletions
|
|
@ -11,11 +11,12 @@ redirect_from: /docs/reference/execution-providers/DirectML-ExecutionProvider
|
|||
|
||||
The DirectML Execution Provider is a component of ONNX Runtime that uses [DirectML](https://docs.microsoft.com/en-us/windows/ai/directml/dml-intro) to accelerate inference of ONNX models. The DirectML execution provider is capable of greatly improving evaluation time of models using commodity GPU hardware, without sacrificing broad hardware support or requiring vendor-specific extensions to be installed.
|
||||
|
||||
|
||||
DirectML is a high-performance, hardware-accelerated DirectX 12 library for machine learning on Windows. DirectML provides GPU acceleration for common machine learning tasks across a broad range of supported hardware and drivers.
|
||||
|
||||
When used standalone, the DirectML API is a low-level DirectX 12 library and is suitable for high-performance, low-latency applications such as frameworks, games, and other real-time applications. The seamless interoperability of DirectML with Direct3D 12 as well as its low overhead and conformance across hardware makes DirectML ideal for accelerating machine learning when both high performance is desired, and the reliability and predictability of results across hardware is critical.
|
||||
|
||||
The DirectML Execution Provider currently uses DirectML version [1.8.2](https://www.nuget.org/packages/Microsoft.AI.DirectML/1.8.2) and supports up to ONNX opset 12 ([ONNX v1.7](https://github.com/onnx/onnx/releases/tag/v1.7.0)). Evaluating models which require a higher opset version is unsupported and will yield poor performance.
|
||||
The DirectML Execution Provider currently uses DirectML version [1.9.0](https://www.nuget.org/packages/Microsoft.AI.DirectML/1.9.0) and supports up to ONNX opset 12 ([ONNX v1.7](https://github.com/onnx/onnx/releases/tag/v1.7.0)). Evaluating models which require a higher opset version is unsupported and will yield poor performance.
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
|
@ -24,25 +25,29 @@ The DirectML Execution Provider currently uses DirectML version [1.8.2](https://
|
|||
{:toc}
|
||||
|
||||
## Install
|
||||
Pre-built packages of ORT with the DirectML EP is published on Nuget.org. See: [Install ORT](../install).
|
||||
|
||||
Pre-built packages of ORT with the DirectML EP is published on Nuget.org. See: [Install ONNX Runtime](../install#cccwinml-installs).
|
||||
|
||||
## Requirements
|
||||
|
||||
The DirectML execution provider requires any DirectX 12 capable device. Almost all commercially-available graphics cards released in the last several years support DirectX 12. Examples of compatible hardware include:
|
||||
The DirectML execution provider requires a DirectX 12 capable device. Almost all commercially-available graphics cards released in the last several years support DirectX 12. Here are some examples of compatible hardware:
|
||||
|
||||
* NVIDIA Kepler (GTX 600 series) and above
|
||||
* AMD GCN 1st Gen (Radeon HD 7000 series) and above
|
||||
* Intel Haswell (4th-gen core) HD Integrated Graphics and above
|
||||
* Qualcomm Adreno 600 and above
|
||||
|
||||
DirectML is compatible with Windows 10, version 1709 (10.0.16299; RS3, "Fall Creators Update") and newer.
|
||||
DirectML was introduced in Windows 10, version 1903, and in the corresponding version of the [Windows SDK](https://docs.microsoft.com/en-us/windows/ai/directml/dml).
|
||||
|
||||
## Build
|
||||
|
||||
Requirements for building the DirectML execution provider:
|
||||
|
||||
1. Visual Studio 2017 toolchain
|
||||
2. [The Windows 10 SDK (10.0.18362.0) for Windows 10, version 1903](https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk) (or newer)
|
||||
|
||||
To build onnxruntime with the DML EP included, supply the `--use_dml` parameter to `build.bat`. e.g.
|
||||
To build onnxruntime with the DML EP included, supply the `--use_dml` flag to `build.bat`.
|
||||
For example:
|
||||
|
||||
```powershell
|
||||
build.bat --config RelWithDebInfo --build_shared_lib --parallel --use_dml
|
||||
|
|
@ -50,7 +55,8 @@ To build onnxruntime with the DML EP included, supply the `--use_dml` parameter
|
|||
|
||||
The DirectML execution provider supports building for both x64 (default) and x86 architectures.
|
||||
|
||||
Note that building onnxruntime with the DirectML execution provider enabled causes the the DirectML redistributable package to be automatically downloaded as part of the build. Its use is governed by a license whose text may be found as part of the NuGet package.
|
||||
Note that, you can build [ONNX Runtime with DirectML](https://docs.microsoft.com/en-us/windows/ai/windows-ml/onnx-versions). This allows DirectML re-distributable package download automatically as part of the build. Find additional license information on the NuGet docs.
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
|
|
@ -81,7 +87,6 @@ OrtStatus* OrtSessionOptionsAppendExecutionProviderEx_DML(
|
|||
);
|
||||
```
|
||||
|
||||
|
||||
## Configuration Options
|
||||
|
||||
The DirectML execution provider does not support the use of memory pattern optimizations or parallel execution in onnxruntime. When supplying session options during InferenceSession creation, these options must be disabled or an error will be returned.
|
||||
|
|
@ -99,23 +104,26 @@ If creating the onnxruntime InferenceSession object directly, you must set the a
|
|||
Additionally, as the DirectML execution provider does not support parallel execution, it does not support multi-threaded calls to `Run` on the same inference session. That is, if an inference session using the DirectML execution provider, only one thread may call `Run` at a time. Multiple threads are permitted to call `Run` simultaneously if they operate on different inference session objects.
|
||||
|
||||
## Performance Tuning
|
||||
The DirectML execution provider works most efficiently when tensor shapes are known at the time a session is created. This provides a few performance benefits:
|
||||
1) Because constant folding can occur more often, there may be fewer CPU / GPU copies and stalls during evaluations.
|
||||
2) More initialization work occurs when sessions are created rather than during the first evaluation.
|
||||
3) Weights may be pre-processed within DirectML, enabling more efficient algorithms to be used.
|
||||
4) Graph optimization occurs within DirectML. For example, Concat operators may be removed, and more optimal tensor layouts may be used for the input and output of operators.
|
||||
|
||||
Normally when the shapes of model inputs are known during session creation, the shapes for the rest of the model are inferred by OnnxRuntime when a session is created. However if a model input contains a free dimension (such as for batch size), steps must be taken to retain the above performance benefits.
|
||||
The DirectML execution provider works most efficiently when tensor shapes are known at the time a session is created. Here are a few performance benefits:
|
||||
|
||||
In this case, there are three options:
|
||||
- Edit the model to replace an input's free dimension (specified through ONNX using "dim_param") with a fixed size (specified through ONNX using "dim_value").
|
||||
- Specify values of named dimensions within model inputs when creating the session using the OnnxRuntime *AddFreeDimensionOverrideByName* ABI.
|
||||
- Edit the model to ensure that an input's free dimension has a [denotation](https://github.com/onnx/onnx/blob/master/docs/DimensionDenotation.md) (such as "DATA_BATCH," or a custom denotation). Then when creating the session, specify the dimension size for each denotation. This can be done using the OnnxRuntime *AddFreeDimensionOverride* ABI.
|
||||
1. Constant folding can occur more often, reducing the CPU / GPU copies and stalls during evaluations.
|
||||
2. More initialization work occurs when sessions are created rather than during the first evaluation.
|
||||
3. Weights may be pre-processed within DirectML, enabling more efficient algorithms.
|
||||
4. Graph optimization occurs within DirectML. For example, Concat operators may be removed and more optimal tensor layouts may be used for the input and output of operators.
|
||||
|
||||
Normally, when the shapes of model inputs are known during a session creation, the shapes for the rest of the model are inferred by OnnxRuntime when that session is created.
|
||||
|
||||
However, if a model input contains a free dimension (such as for batch size), additional steps must be taken to retain the above performance benefits. These include:
|
||||
|
||||
1. Edit the model to replace an input's free dimension (specified through ONNX using "dim_param") with a fixed size (specified through ONNX using "dim_value").
|
||||
2. Specify values of named dimensions within model inputs when creating the session using the OnnxRuntime *AddFreeDimensionOverrideByName* ABI.
|
||||
3. Edit the model to ensure that an input's free dimension has a [denotation](https://github.com/onnx/onnx/blob/master/docs/DimensionDenotation.md) (such as "DATA_BATCH," or a custom denotation). Then, when creating the session, specify the dimension size for each denotation. This can be done using the OnnxRuntime *AddFreeDimensionOverride* ABI.
|
||||
|
||||
## Samples
|
||||
|
||||
A complete sample of onnxruntime using the DirectML execution provider can be found under [samples/c_cxx/fns_candy_style_transfer](https://github.com/microsoft/onnxruntime/tree/master/samples//c_cxx/fns_candy_style_transfer).
|
||||
A complete sample of onnxruntime using the DirectML execution provider can be found under [samples/c_cxx/fns_candy_style_transfer](https://github.com/microsoft/onnxruntime/tree/master/samples//c_cxx/fns_candy_style_transfer)
|
||||
|
||||
|
||||
## Additional Resources
|
||||
|
||||
|
|
@ -123,3 +131,9 @@ A complete sample of onnxruntime using the DirectML execution provider can be fo
|
|||
* [DMLCreateDevice function](https://docs.microsoft.com/windows/win32/api/directml/nf-directml-dmlcreatedevice)
|
||||
* [ID3D12Device::CreateCommandQueue method](https://docs.microsoft.com/windows/win32/api/d3d12/nf-d3d12-id3d12device-createcommandqueue)
|
||||
* [Direct3D 12 programming guide](https://docs.microsoft.com/windows/win32/direct3d12/directx-12-programming-guide)
|
||||
* [ONNX versions and Windows builds](https://docs.microsoft.com/en-us/windows/ai/windows-ml/onnx-versions)
|
||||
* [Windows Machine Learning](https://docs.microsoft.com/en-us/windows/ai/windows-ml/)
|
||||
* [ONNX models](https://docs.microsoft.com/en-us/windows/ai/windows-ml/get-onnx-model)
|
||||
* [DirectML Github](https://github.com/microsoft/DirectML)
|
||||
|
||||
<p><a href="#">Back to top</a></p>
|
||||
|
|
@ -1,12 +1,13 @@
|
|||
---
|
||||
title: WinRT
|
||||
title: Windows
|
||||
parent: Get Started
|
||||
toc: true
|
||||
description: Get Started with Onnx Runtime with Windows. Windows OS Integration and requirements to install and build ORT for Windows are given.
|
||||
nav_order: 8
|
||||
---
|
||||
|
||||
|
||||
# Get started with ORT for Windows Machine Learning WinRT
|
||||
# Get started with ONNX Runtime for Windows
|
||||
{: .no_toc }
|
||||
|
||||
The ONNX Runtime Nuget package provides the ability to use the full [WinML API](https://docs.microsoft.com/en-us/windows/ai/windows-ml/api-reference).
|
||||
|
|
@ -14,7 +15,7 @@ This allows scenarios such as passing a [Windows.Media.VideoFrame](https://docs.
|
|||
|
||||
The WinML API is a WinRT API that shipped inside the Windows OS starting with build 1809 (RS5) in the Windows.AI.MachineLearning namespace. It embedded a version of the ONNX Runtime.
|
||||
|
||||
In addition to using the in-box version of WinML, WinML can also be installed as an application redistributable package (see [layered architecture](../reference/high-level-design#windows-os-integration) for technical details).
|
||||
In addition to using the in-box version of WinML, WinML can also be installed as an application re-distributable package (see [Direct ML Windows](../execution-providers/DirectML-ExecutionProvider) for technical details).
|
||||
|
||||
## Contents
|
||||
{: .no_toc }
|
||||
|
|
@ -22,9 +23,41 @@ In addition to using the in-box version of WinML, WinML can also be installed as
|
|||
* TOC placeholder
|
||||
{:toc}
|
||||
|
||||
|
||||
## Windows OS integration
|
||||
|
||||
ONNX Runtime is available in Windows 10 versions >= 1809 and all versions of Windows 11. It is embedded inside Windows.AI.MachineLearning.dll and exposed via the WinRT API (WinML for short). It includes the CPU execution provider and the [DirectML execution provider](../execution-providers/DirectML-ExecutionProvider) for GPU support.
|
||||
|
||||
The high level design looks like this:
|
||||
|
||||
|
||||
|
||||
### API choice
|
||||
{: .no_toc }
|
||||
|
||||
You can choose to use either the WinRT API or the C API.
|
||||
|
||||
||WinRT|C API|
|
||||
|--|--|--|
|
||||
|Type system| Integration with Windows RT types| Platform neutral types|
|
||||
|Language support| Language support via WinRT Projections| Language support via per language projections|
|
||||
|Tensorization| Accepts VideoFrames and converts to tensors (support for CPU and GPU)| Accepts tensors|
|
||||
|
||||
### Using the NuGet WinRT API with other C-API distributions
|
||||
{: .no_toc }
|
||||
|
||||
The WinRT API NuGet package is distributed with a specific version of ONNX Runtime, but apps can include their own version of ONNX Runtime (either a [released version](../install/#cccwinml-installs) or [a custom build](../build/)). You may wish to do this to use non-default execution providers.
|
||||
To use your own version of ONNX Runtime, replace onnxruntime.dll with your desired version.
|
||||
|
||||
<p><a href="#">Back to top</a></p>
|
||||
|
||||
## Supported Versions
|
||||
Windows 8.1+
|
||||
|
||||
|
||||
|
||||
<p><a href="#">Back to top</a></p>
|
||||
|
||||
## Builds
|
||||
|
||||
|Artifact|Description|Supported Platforms|
|
||||
|
|
@ -59,3 +92,4 @@ If the OS does not have the runtime you need you can switch to use the redist bi
|
|||
|ORT release 1.4| 3|
|
||||
|
||||
See [here](https://docs.microsoft.com/en-us/windows/ai/windows-ml/onnx-versions) for more about opsets and ONNX version details in Windows OS distributions.
|
||||
<p><a href="#">Back to top</a></p>
|
||||
|
|
@ -27,7 +27,7 @@ ONNX Runtime is not explicitly tested with every variation/combination of enviro
|
|||
|
||||
* Tested with Windows 10 and Windows Server 2019
|
||||
* May be compatible with Windows 7+
|
||||
* Windows Machine Learning ([WinRT](../get-started/with-winrt.md))
|
||||
* Windows Machine Learning ([Windows](../get-started/with-windows.md))
|
||||
* CPU: Windows 8.1+
|
||||
* GPU: Windows 10 1709+
|
||||
|
||||
|
|
|
|||
|
|
@ -78,40 +78,4 @@ different representation if they choose to, but it is their responsibility to co
|
|||
* [Add a new graph transform](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime/core/optimizer/graph_transformer.h)
|
||||
* [Add a new rewrite rule](https://github.com/microsoft/onnxruntime/tree/master/include//onnxruntime/core/optimizer/rewrite_rule.h)
|
||||
|
||||
## Windows OS integration
|
||||
|
||||
ONNX Runtime is available in Windows 10 versions >= 1809 and all versions of Windows 11. It is embedded inside Windows.AI.MachineLearning.dll and exposed via the WinRT API (WinML for short). It includes the CPU execution provider and the [DirectML execution provider](../execution-providers/DirectML-ExecutionProvider) for GPU support.
|
||||
|
||||
The high level design looks like this:
|
||||
|
||||
|
||||
|
||||
### API choice
|
||||
{: .no_toc }
|
||||
|
||||
You can choose to use either the WinRT API or the C API.
|
||||
|
||||
||WinRT|C API|
|
||||
|--|--|--|
|
||||
|Type system| Integration with Windows RT types| Platform neutral types|
|
||||
|Language support| Language support via WinRT Projections| Language support via per language projections|
|
||||
|Tensorization| Accepts VideoFrames and converts to tensors (support for CPU and GPU)| Accepts tensors|
|
||||
|
||||
### Distribution choice
|
||||
{: .no_toc }
|
||||
|
||||
You can choose to use the runtime included in Windows, or use the [NuGet package](https://docs.microsoft.com/en-us/nuget/what-is-nuget) to ship the runtime with the app.
|
||||
|
||||
|Distribution|Inbox|App NuGet|
|
||||
|--|--|--|
|
||||
|Disk footprint| Included in the OS| Included in the App|
|
||||
|Servicing fixes| Serviced by OS updates| Serviced by the App|
|
||||
|Execution Providers| CPU & DirectML EP | App chosen EP|
|
||||
|Compatibility testing| Tested with OS flights against supported GPUs and CPU's | App performs compatibility testing|
|
||||
|Opset| Refreshed in OS updates| App chooses|
|
||||
|
||||
### Using the NuGet WinRT API with other C-API distributions
|
||||
{: .no_toc }
|
||||
|
||||
The WinRT API NuGet package is distributed with a specific version of ONNX Runtime, but apps can include their own version of ONNX Runtime (either a [released version](../install/#cccwinml-installs) or [a custom build](../build/)). You may wish to do this to use non-default execution providers.
|
||||
To use your own version of ONNX Runtime, replace onnxruntime.dll with your desired version.
|
||||
<p><a href="#">Back to top</a></p>
|
||||
Loading…
Reference in a new issue