[Vitis AI](https://github.com/Xilinx/Vitis-AI) is AMD's development stack for hardware-accelerated AI inference on AMD platforms, including Ryzen AI, AMD Adaptable SoCs and Alveo Data Center Acceleration Cards. It consists of optimized IP, tools, libraries, models, and example designs.
This release of the Vitis AI Execution Provider enables acceleration of Neural Network model inference targeting AMD Ryzen AI and Adaptable SoCs and Ryzen AI.
A [demonstration](https://github.com/amd/RyzenAI-SW/tree/main/demo/cloud-to-client) is available that showcases one potential application of AMD's XDNA technology. New users are encouraged to test it out.
For AMD Adaptable SoC targets, a pre-built package is provided to deploy ONNX models on embedded Linux. Users should refer to the standard Vitis AI [Target Setup Instructions](https://xilinx.github.io/Vitis-AI/3.5/html/docs/workflow.html) to enable Vitis AI on the target. Once Vitis AI has been enabled on the target, the developer can refer to [this section](https://docs.xilinx.com/r/en-US/ug1414-vitis-ai/Programming-with-VOE) of the Vitis AI documentation for installation and API details.
For more complete examples, developers should refer to [ONNX Runtime Vitis AI Execution Provider examples](https://github.com/Xilinx/Vitis-AI/tree/master/examples/vai_library/samples_onnx).
### Ryzen AI Installation
To enable the Vitis AI ONNX Runtime Execution Provider in Microsoft Windows, a .zip archive is provided.
The developers can refer to the installation section of [Ryzen AI Documentation](https://ryzenai.docs.amd.com/en/latest/inst.html) for more detailed instruction.
**_NOTE:_** Ryzen AI Linux support is not enabled in this release.
Both C++ and Python APIs are supported. The following instructions assume that you have already installed ONNX Runtime on your Windows RyzenAI target. The steps to install the Vitis AI ONNX Runtime Engine are as follows:
- Download the [Ryzen AI Software Package](https://account.amd.com/en/forms/downloads/ryzen-ai-software-platform-xef.html?filename=ryzen-ai-sw-1.0.zip).
- Unzip `ryzen-ai-sw-1.0.zip`.
- Enter the `voe-4.0-win_amd64` ONNX runtime folder.
- Copy the DLL files from the `voe-0.1.0-cp39-cp39-win_amd64` subdirectory of the extracted archive to `C:\Program Files\onnxruntime\bin`, (ie `copy voe-0.1.0-cp39-cp39-win_amd64\*.dll C:\Program Files\onnxruntime\bin`). This will install the Vitis AI ONNX Runtime Engine libraries.
**4. Set the XLNX_VART_FIRMWARE environmental variable:**
**_IMPORTANT:_** If you do not execute this step, the IPU will not be loaded with the required executable file. By default the search path used will be `C:\Windows\System32\AMD` and this path may already contain an xclbin that was stored during the IPU driver installation. Do not use this path to store the Execution Provider xclbin file. Execute the following command, replacing `[path_to_xclbin]` with the target path containing the xclbin:
```
set XLNX_VART_FIRMWARE=C:\[path_to_xclbin]\1x4.xclbin
```
**4. Install the Python Modules:**
- From the Python prompt, execute the following command, replacing `[version]` with the appropriate release version number as determined through the files extracted from the .zip archive.
AMD Ryzen AI and Adaptable SoC targets employ INT8 quantization for inference. The benefits of INT8 quantization include reduced power consumption and memory utilization.
The current release of the Vitis AI Execution Provider ingests quantized ONNX models with INT8 datatypes. No support is provided for direct deployment of models with alternative datatypes, including FP32. Quantization of Ryzen AI models can be accomplished with either the Vitis AI Quantizer, or Olive. For AMD Adapable SoCs, the user must leverage the Vitis AI Quantizer.
[Pytorch](https://hub.docker.com/r/amdih/ryzen-ai-pytorch), [Tensorflow 2.x](https://hub.docker.com/r/amdih/ryzen-ai-tensorflow2) and [Tensorflow 1.x](https://hub.docker.com/r/amdih/ryzen-ai-tensorflow) dockers are avaialable to support quantization of PyTorch and TensorFlow models. To support the Vitis AI ONNX Runtime Execution Provider, an option is provided in the Vitis AI Quantizer to export a quantized model in ONNX format, post quantization.
**ONNX Quantizer python wheel** is available to parse and quantize ONNX models, enabling an end-to-end ONNX model -> ONNX Runtime workflow which is provided in the [Ryzen AI Software Package](https://account.amd.com/en/forms/downloads/ryzen-ai-software-platform-xef.html?filename=ryzen-ai-sw-1.0.zip) as well. Also, in a future release, the Vitis AI ONNX Runtime Execution Provider will support on-the-fly quantization, enabling direct deployment of FP32 ONNX models.
Experimental support for Microsoft Olive is enabled in this release. The Vitis AI Quantizer has been integrated as a plugin into Olive and will be upstreamed. Once this is complete, users can refer to the example(s) provided in the [Olive Vitis AI Example Directory](https://github.com/microsoft/Olive/tree/main/examples/resnet).
The Vitis AI ONNX Runtime integrates a compiler that compiles the model graph and weights as a micro-coded executable. This executable is deployed on the target accelerator (Ryzen AI IPU or Vitis AI DPU).
The model is compiled when the ONNX Runtime session is started, and compilation must complete prior to the first inference pass. The length of time required for compilation varies, but may take a few minutes to complete. Once the model has been compiled, the model executable is cached and for subsequent inference runs, the cached executable model can optionally be used (details below).
Several runtime variables can be set to configure the inference session as listed in the table below. The `config file` variable is not optional and must be set to point to the location of the configuration file. The `cacheDir` and `cacheKey` variables are optional. An example illustrating the usage of all three variables can be found in the C++ [Ryzen AI API Example](#ryzen-ai-api-examples).
| config_file | "" | required, the configuration file path, the configuration file `vaip_config.json` is contained in the `voe-[version]-win_amd64.zip`. |
| XLNX_VART_FIRMWARE | "" | Configures the path location for the xclbin executable file that runs on the IPU. It is essential to configure this variable.|
| XLNX_ENABLE_CACHE | 1 | Whether to use cache, if it is 0, it will ignore the cached executable and the model will be recompiled.|
| XLNX_TARGET_NAME | "" | DPU target name. On Adaptable SoCs, if not set, the DPU target name will be read automatically; On Windows, default value is "AMD_AIE2_Nx4_Overlay" which is the DPU target name of the IPU. |
## Ryzen AI API Examples
To leverage the C++ APIs, use the following example as a reference:
