saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-14 18:12:05 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

update doc for Vitis-AI execution provider upgrade (#15821)

Browse source 
### Description
Documentation and Vitis AI EP synchronous upgrade



### Motivation and Context

---------

Co-authored-by: mingyue <mingyue@amd.com>
This commit is contained in:
mingyueliuh 2023-05-20 00:16:53 +08:00 committed by GitHub
parent 89d77df4cc
commit efbf2d0810
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
3 changed files with 203 additions and 93 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

35
docs/build/eps.md vendored
View file

@ -608,23 +608,34 @@ set(CMAKE_C_COMPILER aarch64-linux-gnu-gcc)
---
## Vitis-AI
See more information on the Xilinx Vitis-AI execution provider [here](../execution-providers/community-maintained/Vitis-AI-ExecutionProvider.md).
## AMD Vitis AI
See more information on the Vitis AI Execution Provider [here](../execution-providers/community-maintained/Vitis-AI-ExecutionProvider.md).
For instructions to setup the hardware environment: [Hardware setup](../execution-providers/community-maintained/Vitis-AI-ExecutionProvider.md#hardware-setup)
### Windows
{: .no_toc }
From the Visual Studio Developer Command Prompt or Developer PowerShell, execute the following command:
```
.\build.bat --use_vitisai --build_shared_lib --parallel --config Release
```
If you wish to leverage the Python APIs, please include the `--build_wheel` flag:
```
.\build.bat --use_vitisai --build_shared_lib --parallel --config Release --build_wheel
```
You can override also override the installation location by specifying CMAKE_INSTALL_PREFIX via the cmake_extra_defines parameter.
e.g.
```
.\build.bat --use_vitisai --build_shared_lib --parallel --config Release --cmake_extra_defines CMAKE_INSTALL_PREFIX=D:\onnxruntime
```
### Linux
{: .no_toc }
```bash
./build.sh --use_vitisai
```
### Notes
{: .no_toc }
The Vitis-AI execution provider is only supported on Linux.
Currently Linux support is only enabled for AMD Adapable SoCs. Please refer to the guidance [here](../execution-providers/community-maintained/Vitis-AI-ExecutionProvider.md#amd-adaptable-soc-installation) for SoC targets.
---

251
docs/execution-providers/community-maintained/Vitis-AI-ExecutionProvider.md
View file

@ -1,18 +1,18 @@
---
title: Xilinx - Vitis AI
description: Instructions to execute ONNX Runtime on Xilinx devices with the Vitis AI execution provider
title: AMD - Vitis AI
description: Instructions to execute ONNX Runtime on AMD devices with the Vitis AI execution provider
grand_parent: Execution Providers
parent: Community-maintained
nav_order: 6
redirect_from: /docs/reference/execution-providers/Vitis-AI-ExecutionProvider
---
# Vitis-AI Execution Provider
# Vitis AI Execution Provider
{: .no_toc }
[Vitis-AI](https://github.com/Xilinx/Vitis-AI) is Xilinx's development stack for hardware-accelerated AI inference on Xilinx platforms, including both edge devices and Alveo cards. It consists of optimized IP, tools, libraries, models, and example designs. It is designed with high efficiency and ease of use in mind, unleashing the full potential of AI acceleration on Xilinx FPGA and ACAP.
[Vitis AI](https://github.com/Xilinx/Vitis-AI) is AMD's development stack for hardware-accelerated AI inference on AMD platforms, including Ryzen AI, AMD Adaptable SoCs and Alveo Data Center Acceleration Cards. It consists of optimized IP, tools, libraries, models, and example designs.
The current Vitis-AI execution provider inside ONNXRuntime enables acceleration of Neural Network model inference using DPUv1. DPUv1 is a hardware accelerator for Convolutional Neural Networks (CNN) on top of the Xilinx [Alveo](https://www.xilinx.com/products/boards-and-kits/alveo.html) platform and targets U200 and U250 accelerator cards.
This release of the Vitis AI Execution Provider enables acceleration of Neural Network model inference targeting AMD Ryzen AI and Adaptable SoCs and Ryzen AI.
## Contents
@ -23,106 +23,199 @@ The current Vitis-AI execution provider inside ONNXRuntime enables acceleration
## Requirements
The following table lists system requirements for running docker containers as well as Alveo cards.
The following table lists AMD targets that are supported by the Vitis AI ONNX Runtime Execution Provider.
| **Architecture** | **Family** | **Supported Targets** | **Supported OS** |
|---------------------------------------------------|------------------------------------------------------------|------------------------------------------------------------|------------------------------------------------------------|
| AMD64 | Ryzen AI | AMD Ryzen 7040U, 7040HS | Windows |
| ARM64 Cortex-A53 | Zynq UltraScale+ MPSoC | ZCU102, ZCU104, KV260 | Linux |
| ARM64 Cortex-A72 | Versal AI Core / Premium | VCK190 | Linux |
| ARM64 Cortex-A72 | Versal AI Edge | VEK280 | Linux |
| **Component** | **Requirement** |
|-----------------------------------------------------|------------------------------------------------------------|
| Motherboard | PCI Express 3\.0\-compliant with one dual\-width x16 slot |
| System Power Supply | 225W |
| Operating System | Ubuntu 16\.04, 18\.04 |
| | CentOS 7\.4, 7\.5 |
| | RHEL 7\.4, 7\.5 |
| CPU | Intel i3/i5/i7/i9/Xeon 64-bit CPU |
| GPU \(Optional to accelerate quantization\) | NVIDIA GPU with a compute capability > 3.0 |
| CUDA Driver \(Optional to accelerate quantization\) | nvidia\-410 |
| FPGA | Xilinx Alveo U200 or U250 |
| Docker Version | 19\.03\.1 |
AMD Adaptable SoC developers can also leverage the Vitis AI ONNX Runtime Execution Provider to support custom (chip-down) designs.
## Ryzen AI Demo
A [demonstration](https://github.com/amd/ms-build-demo) is available that showcases one potential application of AMD's XDNA technology. New users are encouraged to test it out.
## Install
### AMD Adaptable SoC Installation
For AMD Adaptable SoC targets, a pre-built package is provided to deploy ONNX models on embedded Linux. Users should refer to the standard Vitis AI [Target Setup Instructions](https://xilinx.github.io/Vitis-AI/docs/board_setup/board_setup.html) to enable Vitis AI on the target. Once Vitis AI has been enabled on the target, the developer can refer to [this section](https://docs.xilinx.com/r/en-US/ug1414-vitis-ai/Programming-with-VOE) of the Vitis AI documentation for installation and API details.
For more complete examples, developers should refer to [ONNX Runtime Vitis AI Execution Provider examples](https://github.com/Xilinx/Vitis-AI/tree/master/examples/vai_library/samples_onnx).
### Ryzen AI Installation
To enable the Vitis AI ONNX Runtime Execution Provider in Microsoft Windows, a .zip archive is provided.
The contents of this archive are as follows:
```
voe-3.0-win_amd64.zip
│
├── Examples
│ │
│ └── resnet50_python # ONNX ResNet50 ILSVRC2012
│
├── Install
│ │
│ ├── bin # Folder containing DLLs for C++ API
│ ├── 1x4.xclbin # IPU Execution Provider executable file
│ ├── vaip_config.json # Runtime configuration file
│ └── voe-0.1.0-cp39-cp39-win_amd64.whl # Python installation package
```
**_NOTE:_** Ryzen AI Linux support is not enabled in this release.
Both C++ and Python APIs are supported. The following instructions assume that you have already installed ONNX Runtime on your Windows RyzenAI target. The steps to install the Vitis AI ONNX Runtime Engine are as follows:
**1. Verify Pre-requisites:**
- Visual Studio = 2019
- cmake (version >= 3.26)
- python (version >= 3.9) (Python 3.9.13 64bit recommended)
- AMD IPU driver >= 10.105.5.38 installed
**2. Prepare the files for installation:**
- Download the [Ryzen AI ONNX Runtime Package](https://www.xilinx.com/bin/public/openDownload?filename=voe-3.0-win_amd64.zip).
- Unzip `voe-[version]-win_amd64.zip`.
**3. Install the C++ Libraries:**
**_NOTE:_** This is an optional step which is only required if you intend to use the C++ APIs.
- Copy the DLL files from the bin subdirectory of the extracted archive to `C:\Program Files\onnxruntime\bin`, (ie `copy bin\*.dll C:\Program Files\onnxruntime\bin`). This will install the Vitis AI ONNX Runtime Engine libraries.
**4. Set the XLNX_VART_FIRMWARE environmental variable:**
**_IMPORTANT:_** If you do not execute this step, the IPU will not be loaded with the required executable file. By default the search path used will be `C:\Windows\System32\AMD` and this path may already contain an xclbin that was stored during the IPU driver installation. Do not use this path to store the Execution Provider xclbin file. Execute the following command, replacing `[path_to_xclbin]` with the target path containing the xclbin:
```
set XLNX_VART_FIRMWARE=C:\[path_to_xclbin]\1x4.xclbin
```
**4. Install the Python Modules:**
- From the Python prompt, execute the following command, replacing `[version]` with the appropriate release version number as determined through the files extracted from the .zip archive.
```
pip install voe-[version]-cp39-cp39-win_amd64.whl
```
**4. Run the ResNet50 example:**
- Leverage the scripts in the `Examples\resnet50_python` folder to test your installation.
## Build
See [Build instructions](../../build/eps.md#vitis-ai).
To build the Ryzen AI Vitis AI ONNX Runtime Execution Provider from source, please refer to the [Build Instructions](../../build/eps.md#amd-vitis-ai).
### Hardware setup
1. Clone the Vitis AI repository:
```
git clone https://github.com/xilinx/vitis-ai
```
2. Install the Docker, and add the user to the docker group. Link the user to docker installation instructions from the following docker's website:
* https://docs.docker.com/install/linux/docker-ce/ubuntu/
* https://docs.docker.com/install/linux/docker-ce/centos/
* https://docs.docker.com/install/linux/linux-postinstall/
3. Any GPU instructions will have to be separated from Vitis AI.
4. Set up Vitis AI to target Alveo cards. To target Alveo cards with Vitis AI for machine learning workloads, you must install the following software components:
* Xilinx Runtime (XRT)
* Alveo Deployment Shells (DSAs)
* Xilinx Resource Manager (XRM) (xbutler)
* Xilinx Overlaybins (Accelerators to Dynamically Load - binary programming files)
## Quantization
While it is possible to install all of these software components individually, a script has been provided to automatically install them at once. To do so:
* Run the following commands:
```
cd Vitis-AI/alveo/packages
sudo su
./install.sh
```
* Power cycle the system.
5. Build and start the ONNXRuntime Vitis-AI Docker Container.
```
cd {onnxruntime-root}/dockerfiles
docker build -t onnxruntime-vitisai -f Dockerfile.vitisai .
./scripts/docker_run_vitisai.sh
```
Setup inside container
```
source /opt/xilinx/xrt/setup.sh
conda activate vitis-ai-tensorflow
```
AMD Ryzen AI and Adaptable SoC targets employ INT8 quantization for inference. The benefits of INT8 quantization include reduced power consumption and memory utilization.
## Usage
The current release of the Vitis AI Execution Provider ingests quantized ONNX models with INT8 datatypes. No support is provided for direct deployment of models with alternative datatypes, including FP32. Quantization of Ryzen AI models can be accomplished with either the Vitis AI Quantizer, or Olive. For AMD Adapable SoCs, the user must leverage the Vitis AI Quantizer.
### On-the-fly quantization
### Vitis AI Quantizer
In the current release (3.0), the Vitis AI Quantizer supports quantization of PyTorch and TensorFlow models. To support the Vitis AI ONNX Runtime Execution Provider, an option is provided in the Vitis AI Quantizer to export a quantized model in ONNX format, post quantization.
Usually, to be able to accelerate inference of Neural Network models with Vitis-AI DPU accelerators, those models need to quantized upfront. In the ONNXRuntime Vitis-AI execution provider we make use of on-the-fly quantization to remove this additional preprocessing step. In this flow, one doesn't need to quantize his/her model upfront but can make use of the typical inference execution calls (InferenceSession.run) to quantize the model on-the-fly using the first N inputs that are provided (see more information below). This will set up and calibrate the Vitis-AI DPU and from that point onwards inference will be accelerated for all next inputs.
With the future release of Vitis AI 3.5, available mid 2023, the Vitis AI Quantizer will enable parsing and quantization of ONNX models, enabling an end-to-end ONNX model -> ONNX Runtime workflow. Also, in a future release, the Vitis AI ONNX Runtime Execution Provider will support on-the-fly quantization, enabling direct deployment of FP32 ONNX models.
## Configuration Options
See [Vitis AI Model Quantization](https://xilinx.github.io/Vitis-AI/docs/workflow-model-development.html#model-quantization) for details.
A couple of environment variables can be used to customize the Vitis-AI execution provider.
### Olive
| **Environment Variable** | **Default if unset** | **Explanation** |
|----------------------------|---------------------------|---------------------------------------------------------|
| PX_QUANT_SIZE | 128 | The number of inputs that will be used for quantization (necessary for Vitis-AI acceleration) |
| PX_BUILD_DIR | Use the on-the-fly quantization flow | Loads the quantization and compilation information from the provided build directory and immediately starts Vitis-AI hardware acceleration. This configuration can be used if the model has been executed before using on-the-fly quantization during which the quantization and comilation information was cached in a build directory. |
Experimental support for Microsoft Olive is enabled in this release. The Vitis AI Quantizer has been integrated as a plugin into Olive and will be upstreamed. Once this is complete, users can refer to the Vitis AI example(s) provided in the [Olive Vitis AI Example Directory](https://github.com/microsoft/Olive/tree/main/examples/resnet_vitis_ai_ptq_cpu).
## Samples
## Runtime Options
When using python, you can base yourself on the following example:
The Vitis AI ONNX Runtime integrates a compiler that compiles the model graph and weights as a micro-coded executable. This executable is deployed on the target accelerator (Ryzen AI IPU or Vitis AI DPU).
The model is compiled when the ONNX Runtime session is started, and compilation must complete prior to the first inference pass. The length of time required for compilation varies, but may take a few minutes to complete. Once the model has been compiled, the model executable is cached and for subsequent inference runs, the cached executable model can optionally be used (details below).
Several runtime variables can be set to configure the inference session as listed in the table below. The `config file` variable is not optional and must be set to point to the location of the configuration file. The `cacheDir` and `cacheKey` variables are optional. An example illustrating the usage of all three variables can be found in the C++ [Ryzen AI API Example](#ryzen-ai-api-examples).
| **Runtime Variable** | **Default Value** | **Details** |
|----------------------------|--------------------------------|------------------------------------------|
| config_file | "" | required, the configuration file path, the configuration file `vaip_config.json` is contained in the `voe-[version]-win_amd64.zip`. |
| cacheDir | Linux: "/tmp/{user}/vaip/.cache/" <br/> Windows: "C:\\temp\\{user}\\vaip\\.cache" | optional, cache directory |
| cacheKey | {onnx_model_md5} | optional, cache key, used to distinguish between different models. |
The final cache directory is `{cacheDir}/{cacheKey}`.
Please refer to the following C++ example for usage.
In addition, environment variables can be set in order to customize the Vitis AI Execution provider.
| **Environment Variable** | **Default Value** | **Details** |
|----------------------------|----------------------|----------------------------------------------------|
| XLNX_VART_FIRMWARE | "" | Configures the path location for the xclbin executable file that runs on the IPU. It is essential to configure this variable.|
| XLNX_ENABLE_CACHE | 1 | Whether to use cache, if it is 0, it will ignore the cached executable and the model will be recompiled.|
| XLNX_TARGET_NAME | "" | DPU target name. On Adaptable SoCs, if not set, the DPU target name will be read automatically; On Windows, default value is "AMD_AIE2_Nx4_Overlay" which is the DPU target name of the IPU. |
## Ryzen AI API Examples
To leverage the C++ APIs, use the following example as a reference:
```c++
// ...
#include <experimental_onnxruntime_cxx_api.h>
// include user header files
// ...
auto onnx_model_path = "resnet50_pt.onnx"
Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "resnet50_pt");
auto session_options = Ort::SessionOptions();
auto options = std::unorderd_map<std::string,std::string>({});
options["config_file"] = "/etc/vaip_config.json";
// optional, eg: cache path : /tmp/my_cache/abcdefg // Replace abcdefg with your model name, eg. onnx_model_md5
options["cacheDir"] = "/tmp/my_cache";
options["cacheKey"] = "abcdefg"; // Replace abcdefg with your model name, eg. onnx_model_md5
// Create an inference session using the Vitis AI execution provider
session_options.AppendExecutionProvider("VitisAI", options);
auto session = Ort::Experimental::Session(env, model_name, session_options);
auto input_shapes = session.GetInputShapes();
// preprocess input data
// ...
// Create input tensors and populate input data
std::vector<Ort::Value> input_tensors;
input_tensors.push_back(Ort::Experimental::Value::CreateTensor<float>(
input_data.data(), input_data.size(), input_shapes[0]));
auto output_tensors = session.Run(session.GetInputNames(), input_tensors,
session.GetOutputNames());
// postprocess output data
// ...
```
To leverage the Python APIs, use the following example as a reference:
```python
# Import pyxir before onnxruntime
import pyxir
import pyxir.frontend.onnx
import pyxir.contrib.dpuv1.dpuv1
import onnxruntime
# Add other imports
# Add user imports
# ...
# Load inputs and do preprocessing
# ...
# Create an inference session using the Vitis-AI execution provider
session = onnxruntime.InferenceSession('[model_file].onnx', None,["VitisAIExecutionProvider"])
# Create an inference session using the Vitis AI execution provider
session = onnxruntime.InferenceSession(
'[model_file].onnx',
providers=["VitisAIExecutionProvider"],
provider_options=[{"config_file":"/etc/vaip_config.json"}])
# First N (default = 128) inputs are used for quantization calibration and will
# be executed on the CPU
# This config can be changed by setting the 'PX_QUANT_SIZE' (e.g. export PX_QUANT_SIZE=64)
imput_name = [...]
outputs = [session.run([], {input_name: calib_inputs[i]})[0] for i in range(128)]
input_shape = session.get_inputs()[0].shape
input_name = session.get_inputs()[0].name
# Afterwards, computations will be accelerated on the FPGA
# Load inputs and do preprocessing by input_shape
input_data = [...]
result = session.run([], {input_name: input_data})
```

10
js/script.js
View file

@ -1033,10 +1033,16 @@ var validCombos = {
"ios,C#,ARM64,CoreML":
"Install Nuget package&nbsp;<a href='https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime' target='_blank'>Microsoft.ML.OnnxRuntime</a>.",
"windows,Python,X86,VitisAI":
"windows,Python,X64,VitisAI":
"Follow build instructions from <a href='https://aka.ms/build-ort-vitisai' target='_blank'>here</a>",
"linux,Python,X86,VitisAI":
"windows,C++,X64,VitisAI":
"Follow build instructions from <a href='https://aka.ms/build-ort-vitisai' target='_blank'>here</a>",
"linux,C++,ARM64,VitisAI":
"Follow build instructions from <a href='https://aka.ms/build-ort-vitisai' target='_blank'>here</a>",
"linux,Python,ARM64,VitisAI":
"Follow build instructions from <a href='https://aka.ms/build-ort-vitisai' target='_blank'>here</a>",
"linux,Python,X64,MIGraphX":