From f7b4bc15e1d0120a68bc3f8909c6b002a9a5d1a2 Mon Sep 17 00:00:00 2001 From: suryasidd <48925384+suryasidd@users.noreply.github.com> Date: Thu, 24 Oct 2019 14:31:44 -0700 Subject: [PATCH] Updated documentation for VAD-F (#2248) Signed-off-by: suryasidd --- BUILD.md | 31 ++--- .../OpenVINO-ExecutionProvider.md | 117 +++++++++--------- 2 files changed, 76 insertions(+), 72 deletions(-) diff --git a/BUILD.md b/BUILD.md index 86be1005fc..259f20cfb4 100644 --- a/BUILD.md +++ b/BUILD.md @@ -117,7 +117,7 @@ The complete list of build options can be found by running `./build.sh (or .\bui Read more about ONNX Runtime Server [here](https://github.com/microsoft/onnxruntime/blob/master/docs/ONNX_Runtime_Server_Usage.md) ### Pre-Requisites -* ONNX Runtime server (and only the server) requires you to have Go installed to build, due to building BoringSSL. +* ONNX Runtime server (and only the server) requires you to have Go installed to build, due to building BoringSSL. See https://golang.org/doc/install for installation instructions. ### Build Instructions @@ -127,7 +127,7 @@ Read more about ONNX Runtime Server [here](https://github.com/microsoft/onnxrunt ONNX Runtime Server supports sending logs to [rsyslog](https://www.rsyslog.com/) daemon. To enable it, please build with an additional parameter: `--cmake_extra_defines onnxruntime_USE_SYSLOG=1`. -Build command: +Build command: ``` ./build.sh --config RelWithDebInfo --build_server --use_openmp --parallel --cmake_extra_defines onnxruntime_USE_SYSLOG=1 ``` @@ -159,7 +159,7 @@ Build command: A Dockerfile is available [here](./tools/ci_build/github/linux/docker/Dockerfile.ubuntu_gpu). - + #### Notes * Depending on compatibility between the CUDA, cuDNN, and Visual Studio 2017 versions you are using, you may need to explicitly install an earlier version of the MSVC toolset. * CUDA 10.0 is [known to work](https://devblogs.microsoft.com/cppblog/cuda-10-is-now-available-with-support-for-the-latest-visual-studio-2017-versions/) with toolsets from 14.11 up to 14.16 (Visual Studio 2017 15.9), and should continue to work with future Visual Studio versions @@ -169,7 +169,7 @@ A Dockerfile is available [here](./tools/ci_build/github/linux/docker/Dockerfile 1. Setup the Visual Studio environment variables to point to the 14.11 toolset by running vcvarsall.bat, prior to running the build script. e.g. if you have VS2017 Enterprise, an x64 build would use the following command `"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" amd64 -vcvars_ver=14.11` For convenience, .\build.amd64.1411.bat will do this and can be used in the same way as .\build.bat. e.g. ` .\build.amd64.1411.bat --use_cuda` 2. Alternatively, if you have CMake 3.12 or later you can specify the toolset version via the `--msvc_toolset` build script parameter. e.g. `.\build.bat --msvc_toolset 14.11` - + * If you have multiple versions of CUDA installed on a Windows machine and are building with Visual Studio, CMake will use the build files for the highest version of CUDA it finds in the BuildCustomization folder. e.g. C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\VC\VCTargets\BuildCustomizations\. If you want to build with an earlier version, you must temporarily remove the 'CUDA x.y.*' files for later versions from this directory. @@ -182,14 +182,14 @@ If you want to build with an earlier version, you must temporarily remove the 'C See more information on the TensorRT Execution Provider [here](./docs/execution_providers/TensorRT-ExecutionProvider.md). #### Pre-Requisites -* Install [CUDA](https://developer.nvidia.com/cuda-toolkit) and [cuDNN](https://developer.nvidia.com/cudnn) +* Install [CUDA](https://developer.nvidia.com/cuda-toolkit) and [cuDNN](https://developer.nvidia.com/cudnn) * The TensorRT execution provider for ONNX Runtime is built and tested with CUDA 10.1 and cuDNN 7.6. * The path to the CUDA installation must be provided via the CUDA_PATH environment variable, or the `--cuda_home parameter`. The CUDA path should contain `bin`, `include` and `lib` directories. * The path to the CUDA `bin` directory must be added to the PATH environment variable so that `nvcc` is found. * The path to the cuDNN installation (path to folder that contains libcudnn.so) must be provided via the cuDNN_PATH environment variable, or `--cudnn_home parameter`. * Install [TensorRT](https://developer.nvidia.com/nvidia-tensorrt-download) * The TensorRT execution provider for ONNX Runtime is built and tested with TensorRT 6.0.1.5 but validated with the feature set equivalent to TensorRT 5. Some TensorRT 6 new features such as dynamic shape is not available at this time. - * The path to TensorRT installation must be provided via the `--tensorrt_home parameter`. + * The path to TensorRT installation must be provided via the `--tensorrt_home parameter`. #### Build Instructions ##### Linux @@ -244,6 +244,7 @@ See more information on the OpenVINO Execution Provider [here](./docs/execution_ * To configure Intel® Processor Graphics(GPU) please follow these instructions: [Windows](https://docs.openvinotoolkit.org/2019_R1.1/_docs_install_guides_installing_openvino_windows.html#Install-GPU), [Linux](https://docs.openvinotoolkit.org/2019_R1.1/_docs_install_guides_installing_openvino_linux.html#additional-GPU-steps) * To configure Intel® MovidiusTM USB, please follow this getting started guide: [Windows](https://docs.openvinotoolkit.org/2019_R1.1/_docs_install_guides_installing_openvino_windows.html#usb-myriad), [Linux](https://docs.openvinotoolkit.org/2019_R1.1/_docs_install_guides_installing_openvino_linux.html#additional-NCS-steps) * To configure Intel® Vision Accelerator Design based on 8 MovidiusTM MyriadX VPUs, please follow this configuration guide: [Windows](https://docs.openvinotoolkit.org/2019_R1.1/_docs_install_guides_installing_openvino_windows.html#hddl-myriad), [Linux](https://docs.openvinotoolkit.org/2019_R1.1/_docs_install_guides_installing_openvino_linux.html#install-VPU) + * To configure Intel® Vision Accelerator Design with an Intel® Arria® 10 FPGA, please follow this configuration guide: [Linux](https://docs.openvinotoolkit.org/2019_R1.1/_docs_install_guides_VisionAcceleratorFPGA_Configure_2019R1.html) #### Build Instructions @@ -255,20 +256,20 @@ See more information on the OpenVINO Execution Provider [here](./docs/execution_ ##### Linux ``` -./build.sh --config RelWithDebInfo --use_openvino +./build.sh --config RelWithDebInfo --use_openvino ``` For Linux: - + ./build.sh --config RelWithDebInfo --use_openvino For Windows: - + .\build.bat --config RelWithDebInfo --use_openvino - + *Note: The default Windows CMake Generator is Visual Studio 2017, but you can also use the newer Visual Studio 2019 by passing `--cmake_generator "Visual Studio 16 2019"` to `.\build.bat`* - + --use_openvino: Builds the OpenVINO Execution Provider in ONNX Runtime. @@ -303,11 +304,11 @@ For more information on OpenVINO Execution Provider's ONNX Layer support, To --- ### NUPHAR -See more information on the Nuphar Execution Provider [here](./docs/execution_providers/Nuphar-ExecutionProvider.md). +See more information on the Nuphar Execution Provider [here](./docs/execution_providers/Nuphar-ExecutionProvider.md). #### Pre-Requisites * The Nuphar execution provider for ONNX Runtime is built and tested with LLVM 9.0.0. Because of TVM's requirement when building with LLVM, you need to build LLVM from source. To build the debug flavor of ONNX Runtime, you need the debug build of LLVM. - * Windows (Visual Studio 2017): + * Windows (Visual Studio 2017): ``` REM download llvm source code 9.0.0 and unzip to \llvm\source\path, then install to \llvm\install\path cd \llvm\source\path @@ -317,7 +318,7 @@ See more information on the Nuphar Execution Provider [here](./docs/execution_pr msbuild llvm.sln /maxcpucount /p:Configuration=Release /p:Platform=x64 cmake -DCMAKE_INSTALL_PREFIX=\llvm\install\path -DBUILD_TYPE=Release -P cmake_install.cmake ``` - + *Note that following LLVM cmake patch is necessary to make the build work on Windows, Linux does not need to apply the patch.* The patch is to fix the linking warning LNK4199 caused by this [LLVM commit](https://github.com/llvm-mirror/llvm/commit/148f823e4845c9a13faea62e3105abb80b39e4bc) @@ -385,7 +386,7 @@ See more information on the DirectML execution provider [here](./docs/execution_ ``` #### Notes The DirectML execution provider supports building for both x64 and x86 architectures. DirectML is only supported on Windows. - + --- ## Options diff --git a/docs/execution_providers/OpenVINO-ExecutionProvider.md b/docs/execution_providers/OpenVINO-ExecutionProvider.md index e52a6466e1..d769c49c4c 100644 --- a/docs/execution_providers/OpenVINO-ExecutionProvider.md +++ b/docs/execution_providers/OpenVINO-ExecutionProvider.md @@ -9,32 +9,32 @@ For build instructions, please see the [BUILD page](../../BUILD.md#openvino). The table below shows the ONNX layers supported using OpenVINO Execution Provider and the mapping between ONNX layers and OpenVINO layers. The below table also lists the Intel hardware support for each of the layers. CPU refers to Intel® Atom, Core, and Xeon processors. GPU refers to the Intel Integrated Graphics. VPU refers to USB based Intel® MovidiusTM -VPUs as well as Intel® Vision accelerator Design with Intel Movidius TM MyriadX VPU. +VPUs as well as Intel® Vision accelerator Design with Intel Movidius TM MyriadX VPU. FPGA refers to Intel® Vision Accelerator Design with an Intel® Arria® 10 FPGA. -| **ONNX Layers** | **OpenVINO Layers** | **CPU** | **GPU** | **VPU** | -| --- | --- | --- | --- | --- | -| Add | Eltwise (operation=sum) | Yes | Yes | Yes -| AveragePool | Pooling(pool\_method=avg) | Yes | Yes | Yes -| BatchNormalization | Scaleshift (can be fused into Convlution or Fully Connected) | Yes | Yes | Yes -| Concat | Concat | Yes | Yes | Yes -| Conv | Convolution | Yes | Yes | Yes -| Dropout | Ignored | Yes | Yes | Yes -| Flatten | Reshape | Yes | Yes | Yes -| Gemm | FullyConnected | Yes | Yes | Yes -| GlobalAveragePool | Pooling | Yes | Yes | Yes -| Identity | Ignored | Yes | Yes | Yes -| ImageScaler | ScaleShift | Yes | Yes | Yes -| LRN | Norm | Yes | Yes | Yes -| MatMul | FullyConnected | Yes | Yes* | No -| MaxPool | Pooling(pool\_method=max) | Yes | Yes | Yes -| Mul | Eltwise(operation=mul) | Yes | Yes | Yes -| Relu | ReLU | Yes | Yes | Yes -| Reshape | Reshape | Yes | Yes | Yes -| Softmax | SoftMax | Yes | Yes | Yes -| Sum | Eltwise(operation=sum) | Yes | Yes | Yes -| Transpose | Permute | Yes | Yes | Yes -| UnSqueeze | Reshape | Yes | Yes | Yes -| LeakyRelu | ReLU | Yes | Yes | Yes +| **ONNX Layers** | **OpenVINO Layers** | **CPU** | **GPU** | **VPU** | **FPGA** | +| --- | --- | --- | --- | --- | --- | +| Add | Eltwise (operation=sum) | Yes | Yes | Yes | Yes +| AveragePool | Pooling(pool\_method=avg) | Yes | Yes | Yes | Yes +| BatchNormalization | Scaleshift (can be fused into Convlution or Fully Connected) | Yes | Yes | Yes | Yes +| Concat | Concat | Yes | Yes | Yes | Yes +| Conv | Convolution | Yes | Yes | Yes | Yes +| Dropout | Ignored | Yes | Yes | Yes | Yes +| Flatten | Reshape | Yes | Yes | Yes | No +| Gemm | FullyConnected | Yes | Yes | Yes | Yes +| GlobalAveragePool | Pooling | Yes | Yes | Yes | Yes +| Identity | Ignored | Yes | Yes | Yes | Yes +| ImageScaler | ScaleShift | Yes | Yes | Yes | Yes +| LRN | Norm | Yes | Yes | Yes | Yes +| MatMul | FullyConnected | Yes | Yes* | No | Yes +| MaxPool | Pooling(pool\_method=max) | Yes | Yes | Yes | Yes +| Mul | Eltwise(operation=mul) | Yes | Yes | Yes | No +| Relu | ReLU | Yes | Yes | Yes | Yes +| Reshape | Reshape | Yes | Yes | Yes | No +| Softmax | SoftMax | Yes | Yes | Yes | No +| Sum | Eltwise(operation=sum) | Yes | Yes | Yes | Yes +| Transpose | Permute | Yes | Yes | Yes | No +| UnSqueeze | Reshape | Yes | Yes | Yes | No +| LeakyRelu | ReLU | Yes | Yes | Yes | Yes *MatMul is supported in GPU only when the following layer is an Add layer in the topology. @@ -44,47 +44,50 @@ Below topologies are supported from ONNX open model zoo using OpenVINO Execution ### Image Classification Networks -| **Topology** | **CPU** | **GPU** | **VPU** | -| --- | --- | --- | --- | -| bvlc\_alexnet | Yes | Yes | Yes -| bvlc\_googlenet | Yes | Yes | Yes -| bvlc\_reference\_caffenet | Yes | Yes | Yes -| bvlc\_reference\_rcnn\_ilsvrc13 | Yes | Yes | Yes -| densenet121 | Yes | Yes | Yes -| Inception\_v1 | Yes | Yes | Yes** -| Inception\_v2 | Yes | Yes | Yes -| Shufflenet | Yes | Yes | Yes -| Zfnet512 | Yes | Yes | Yes -| Squeeznet 1.1 | Yes | Yes | Yes -| Resnet18v1 | Yes | Yes | Yes -| Resnet34v1 | Yes | Yes | Yes -| Resnet50v1 | Yes | Yes | Yes -| Resnet101v1 | Yes | Yes | Yes -| Resnet152v1 | Yes | Yes | Yes -| Resnet18v2 | Yes | Yes | Yes -| Resnet34v2 | Yes | Yes | Yes -| Resnet50v2 | Yes | Yes | Yes -| Resnet101v2 | Yes | Yes | Yes -| Resnet152v2 | Yes | Yes | Yes -| Mobilenetv2 | Yes | Yes | Yes -| vgg16 | Yes | Yes | Yes -| vgg19 | Yes | Yes | Yes +| **Topology** | **CPU** | **GPU** | **VPU** | **FPGA** | +| --- | --- | --- | --- | --- | +| bvlc\_alexnet | Yes | Yes | Yes | Yes*** +| bvlc\_googlenet | Yes | Yes | Yes | Yes*** +| bvlc\_reference\_caffenet | Yes | Yes | Yes | Yes*** +| bvlc\_reference\_rcnn\_ilsvrc13 | Yes | Yes | Yes | Yes*** +| densenet121 | Yes | Yes | Yes | Yes*** +| Inception\_v1 | Yes | Yes | Yes** | Yes*** +| Inception\_v2 | Yes | Yes | Yes | Yes*** +| Shufflenet | Yes | Yes | Yes | Yes*** +| Zfnet512 | Yes | Yes | Yes | Yes*** +| Squeeznet 1.1 | Yes | Yes | Yes | Yes*** +| Resnet18v1 | Yes | Yes | Yes | Yes*** +| Resnet34v1 | Yes | Yes | Yes | Yes*** +| Resnet50v1 | Yes | Yes | Yes | Yes*** +| Resnet101v1 | Yes | Yes | Yes | Yes*** +| Resnet152v1 | Yes | Yes | Yes | Yes*** +| Resnet18v2 | Yes | Yes | Yes | Yes*** +| Resnet34v2 | Yes | Yes | Yes | Yes*** +| Resnet50v2 | Yes | Yes | Yes | Yes*** +| Resnet101v2 | Yes | Yes | Yes | Yes*** +| Resnet152v2 | Yes | Yes | Yes | Yes*** +| Mobilenetv2 | Yes | Yes | Yes | Yes*** +| vgg16 | Yes | Yes | Yes | Yes*** +| vgg19 | Yes | Yes | Yes | Yes*** + ### Image Recognition Networks -| **Topology** | **CPU** | **GPU** | **VPU** | -| --- | --- | --- | --- | -| MNIST | Yes | Yes | Yes** +| **Topology** | **CPU** | **GPU** | **VPU** | **FPGA** | +| --- | --- | --- | --- | --- | +| MNIST | Yes | Yes | Yes** | Yes*** **Inception_v1 and MNIST are supported in OpenVINO R1.1 and are not supported in OpenVINO R5.0.1. ### Object Detection Networks -| **Topology** | **CPU** | **GPU** | **VPU** | -| --- | --- | --- | --- | -|TinyYOLOv2 | Yes | Yes | Yes -| ResNet101\_DUC\_HDC | Yes | No | No +| **Topology** | **CPU** | **GPU** | **VPU** | **FPGA** | +| --- | --- | --- | --- | -- | +|TinyYOLOv2 | Yes | Yes | Yes | Yes*** +| ResNet101\_DUC\_HDC | Yes | No | No | Yes*** + +***FPGA only runs in HETERO mode wherein the layers that are not supported on FPGA fall back to OpenVINO CPU. ## Application code changes for VAD-M performance scaling