"**Deploying Facial Emotion Recognition (FER+) using Docker Images for ONNX Runtime with TensorRT**\n",
"\n",
"This example shows how to deploy an image classification neural network using ONNX Runtime on GPU compute SKUs in Azure. This example makes use of the Facial Expression Recognition ([FER](https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data)) dataset and Open Neural Network eXchange format ([ONNX](http://aka.ms/onnxruntime)) on the Azure Machine Learning platform.\n",
"\n",
"\n",
"\n",
"#### Tutorial Roadmap:\n",
"1. Obtain an ***external ONNX model*** for facial emotion recognition (FER+) from the [ONNX model zoo](https://github.com/onnx/models)\n",
"2. ***Register our model*** in our Azure Machine Learning workspace\n",
"3. ***Write a scoring file*** and environment file to evaluate our model with ONNX Runtime\n",
"4. ***Build a container image*** using the ONNX Runtime + TensorRT base image from Microsoft Container Registry (MCR)\n",
"5. ***Deploy to the cloud*** using an AKS cluster with GPU and use it to make predictions using ONNX Runtime Python APIs\n",
"\n",
"**Note:** You can also use the same notebook and code to deploy to a CPU cluster in addition to your GPU cluster. ONNX Runtime APIs remain unchanged across hardware endpoints. The default option for a secondary deployment with CPU is `False`, but you can change the variable below to `True` for a performance comparison."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"deploy_with_cpu = False"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 0. Prerequisites\n",
"\n",
"Throughout this tutorial, we will be referring to **ONNX**, a neural network exchange format used to represent deep learning models. With ONNX, AI developers can more easily move models between state-of-the-art tools (CNTK, PyTorch, Caffe, MXNet, TensorFlow) and choose the combination that is best for them. ONNX is developed and supported by a community of partners including Microsoft AI, Facebook, and Amazon. For more information, explore the [ONNX website](http://onnx.ai) and [open source files](https://github.com/onnx).\n",
"\n",
"[ONNX Runtime](https://aka.ms/onnxruntime-python) is the runtime engine that enables evaluation of trained machine learning (traditional ML and Deep Learning) models with high performance and low resource utilization. We use the CPU version of **ONNX Runtime** in this tutorial, but will soon be releasing an additional tutorial for deploying this model using ONNX Runtime GPU."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Install Azure ML SDK and create a new workspace\n",
"If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, please follow the [Azure ML configuration notebook](https://github.com/Azure/MachineLearningNotebooks/blob/master/configuration.ipynb) to set up your environment.\n",
"\n",
"### Install additional packages needed for this Notebook\n",
"You need to install the popular plotting library matplotlib, the image manipulation library opencv, and the onnx library in the conda environment where Azure Maching Learning SDK is installed.\n",
"**Debugging tip**: Make sure that to activate your virtual environment (myenv) before you re-launch this notebook using the jupyter notebook comand. Choose the respective Python kernel for your new virtual environment using the Kernel > Change Kernel menu above. If you have completed the steps correctly, the upper right corner of your screen should state Python [conda env:myenv] instead of Python [default]."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Obtain a model from the ONNX Model Zoo\n",
"\n",
"For more information on the Facial Emotion Recognition (FER+) model, you can explore the notebook explaning how to deploy [FER+ with ONNX Runtime on an ACI Instance](onnx-inference-facial-expression-recognition-deploy.ipynb)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# urllib is a built-in Python library to download files from URLs\n",
"\n",
"# Objective: retrieve the latest version of the ONNX Emotion FER+ model files from the\n",
"# ONNX Model Zoo and save it in the same folder as this tutorial\n",
"We are now going to deploy our ONNX Model on AKS with inference in ONNX Runtime. We begin by writing a score.py file, which will help us run the model in our Azure Kubernetes Cluster, and then specify our environment by writing a yml file. You will also notice that we import the onnxruntime library to do runtime inference on our ONNX models (passing in input and evaluating out model's predicted output). More information on the API and commands can be found in the [ONNX Runtime documentation](https://aka.ms/onnxruntime).\n",
"\n",
"### Write Score File\n",
"\n",
"A score file is what tells our Azure cloud service what to do. After initializing our model using azureml.core.model, we start an ONNX Runtime inference session to evaluate the data passed in on our function calls."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%writefile score.py\n",
"import json\n",
"import numpy as np\n",
"import onnxruntime\n",
"import sys\n",
"import os\n",
"from azureml.core.model import Model\n",
"import time\n",
"\n",
"def init():\n",
" global session, input_name, output_name\n",
" model = Model.get_model_path(model_name = 'onnx_emotion')\n",
" \n",
" # Load the model in onnx runtime to start the session \n",
" # If your deployment fails, make sure to delete your aks_service before trying again!\n",
" # cpu_aks_service.delete()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Success!\n",
"\n",
"If you've made it this far, you've deployed a working AKS Cluster with a facial emotion recognition model running in the cloud using Azure ML. Congratulations!\n",
"\n",
"Let's see how well our model deals with our test images."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Testing and Evaluation\n",
"\n",
"### Useful Helper Functions\n",
"\n",
"We preprocess and postprocess our data (see score.py file) using the helper functions specified in the [ONNX FER+ Model page in the Model Zoo repository](https://github.com/onnx/models/tree/master/emotion_ferplus)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def emotion_map(classes, N=1):\n",
" \"\"\"Take the most probable labels (output of postprocess) and returns the \n",
" top N emotional labels that fit the picture.\"\"\"\n",
" plt.title(\"ONNX Runtime CPU vs GPU Performance Comparison\")\n",
"\n",
" plt.yticks(x_pos, x)\n",
"\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Clean up our workspace"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# remember to delete your service after you are done using it!\n",
"\n",
"gpu_aks_service.delete()\n",
"\n",
"if deploy_with_cpu:\n",
" cpu_aks_service.delete()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conclusion\n",
"\n",
"Congratulations!\n",
"\n",
"In this tutorial, you have:\n",
"- familiarized yourself with ONNX Runtime inference and the pretrained models in the ONNX model zoo\n",
"- understood a state-of-the-art convolutional neural net image classification model (FER+ in ONNX) and deployed it in the Azure ML cloud\n",
"- ensured that your deep learning model is working perfectly (in the cloud) on test data, and checked it against some of your own!\n",
"\n",
"Next steps:\n",
"- If you have not already, check out another interesting ONNX application that lets you set up a state-of-the-art [handwritten image classification model (MNIST)](https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/deployment/onnx/onnx-inference-mnist-deploy.ipynb) on Azure! This tutorial deploys a pre-trained ONNX Computer Vision model for handwritten digit classification in an Azure ML Container Instance.\n",
"- Contribute to our [open source ONNX repository on github](http://github.com/onnx/onnx) and/or add to our [ONNX model zoo](http://github.com/onnx/models)"
