diff --git a/onnxruntime/python/tools/transformers/notebooks/PyTorch_Bert-Squad_OnnxRuntime_GPU.ipynb b/onnxruntime/python/tools/transformers/notebooks/PyTorch_Bert-Squad_OnnxRuntime_GPU.ipynb index 23138a44d0..f0ad3c189f 100644 --- a/onnxruntime/python/tools/transformers/notebooks/PyTorch_Bert-Squad_OnnxRuntime_GPU.ipynb +++ b/onnxruntime/python/tools/transformers/notebooks/PyTorch_Bert-Squad_OnnxRuntime_GPU.ipynb @@ -19,11 +19,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In this tutorial, you'll be introduced to how to load a Bert model from PyTorch, convert it to ONNX, and inference it for high performance using ONNX Runtime and NVIDIA GPU. In the following sections, we are going to use the Bert model trained with Stanford Question Answering Dataset (SQuAD) dataset as an example. Bert SQuAD model is used in question answering scenarios, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.\n", + "In this tutorial, you'll learn how to load a Bert model from PyTorch, convert it to ONNX, and inference it for high performance using ONNX Runtime and NVIDIA GPU. In the following sections, we are going to use the Bert model trained with Stanford Question Answering Dataset (SQuAD) dataset as an example. Bert SQuAD model is used in question answering scenarios, where the answer to every question is a segment of text from the corresponding reading passage, or the question might be unanswerable.\n", "\n", - "This notebook is for GPU inference. For CPU inference, please look at another notebook [Inference PyTorch Bert Model with ONNX Runtime on CPU](PyTorch_Bert-Squad_OnnxRuntime_CPU.ipynb).\n", - "\n", - "Note that you might need change !{sys.executable} to !python when running the notebook in Linux." + "This notebook is for GPU inference. For CPU inference, please look at another notebook [Inference PyTorch Bert Model with ONNX Runtime on CPU](PyTorch_Bert-Squad_OnnxRuntime_CPU.ipynb)." ] }, { @@ -31,25 +29,38 @@ "metadata": {}, "source": [ "## 0. Prerequisites ##\n", - "It requires your machine to have a GPU, and a python environment with [PyTorch](https://pytorch.org/) and [OnnxRuntime](https://microsoft.github.io/onnxruntime/) installed before running this notebook.\n", + "It requires your machine to have a GPU, and a python environment with [PyTorch](https://pytorch.org/) installed before running this notebook.\n", "\n", "#### GPU Environment Setup using AnaConda\n", "\n", - "First, we install [AnaConda](https://www.anaconda.com/distribution/) in a target machine and open an AnaConda prompt window when it is done. Then run the following commands to create a conda environment. This notebook was run with PyTorch 1.4 and OnnxRuntime 1.2.0. (We also verified it with PyTorch 1.5 and OnnxRuntime 1.3.0).\n", + "First, we install [AnaConda](https://www.anaconda.com/distribution/) in a target machine and open an AnaConda prompt window when it is done. Then run the following commands to create a conda environment. This notebook is tested with PyTorch 1.5.0 and OnnxRuntime 1.3.0.\n", "\n", "```console\n", - "conda create -n gpu_env python=3.6\n", + "conda create -n gpu_env python=3.7\n", "conda activate gpu_env\n", "conda install pytorch torchvision cudatoolkit=10.1 -c pytorch\n", - "pip install onnxruntime-gpu\n", - "pip install transformers==2.11.0\n", - "pip install onnxruntime-tools\n", - "pip install wget netron\n", - "conda install jupyter\n", + "conda install -c anaconda ipykernel\n", + "conda install -c conda-forge ipywidgets\n", + "python -m ipykernel install --user --name=gpu_env_py37\n", "jupyter notebook\n", "```\n", + "Finally, launch Jupyter Notebook and you can choose gpu_env_py37 as kernel to run this notebook.\n", "\n", - "Onnxruntime-gpu need specified version of CUDA and cuDNN. You can find the corresponding version in [release note](https://github.com/microsoft/onnxruntime/releases). If the version is different from above cudatoolkit version, you have to install them separately, and add their bin directories to PATH environment variable (See [CUDA and cuDNN Path](#CUDA-and-cuDNN-Path) below)." + "Onnxruntime-gpu need specified version of CUDA and cuDNN. You can find the corresponding version in [requirements](https://github.com/microsoft/onnxruntime/tree/rel-1.3.0#system-requirements). If the version is different from above cudatoolkit version, you have to install them separately, and add their bin directories to PATH environment variable (See [CUDA and cuDNN Path](#CUDA-and-cuDNN-Path) below)." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import sys\n", + "!{sys.executable} -m pip uninstall --quiet --yes onnxruntime-gpu\n", + "!{sys.executable} -m pip install --quiet onnxruntime-gpu\n", + "!{sys.executable} -m pip install --quiet --upgrade transformers\n", + "!{sys.executable} -m pip install --quiet --upgrade onnxruntime-tools\n", + "!{sys.executable} -m pip install --quiet wget netron pandas" ] }, { @@ -68,7 +79,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 2, "metadata": {}, "outputs": [], "source": [ @@ -96,7 +107,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "metadata": {}, "outputs": [], "source": [ @@ -106,7 +117,7 @@ "# Total samples to inference, so that we can get average latency\n", "total_samples = 1000\n", "\n", - "# ONNX opset version: 10 or 11\n", + "# ONNX opset version\n", "opset_version=11" ] }, @@ -119,7 +130,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 4, "metadata": {}, "outputs": [], "source": [ @@ -139,16 +150,16 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "100%|██████████████████████████████████████████████████████████████████████████████████| 48/48 [00:04<00:00, 10.72it/s]\n", - "convert squad examples to features: 100%|█████████████████████████████████████████| 1000/1000 [00:08<00:00, 123.84it/s]\n", - "add example index and unique id: 100%|█████████████████████████████████████████| 1000/1000 [00:00<00:00, 500274.81it/s]\n" + "100%|██████████| 48/48 [00:04<00:00, 11.28it/s]\n", + "convert squad examples to features: 100%|██████████| 1000/1000 [00:09<00:00, 102.15it/s]\n", + "add example index and unique id: 100%|██████████| 1000/1000 [00:00<00:00, 161306.98it/s]\n" ] } ], @@ -194,14 +205,14 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Model exported at ./onnx\\bert-base-cased-squad_opset11.onnx\n" + "Model exported at ./onnx/bert-base-cased-squad_opset11.onnx\n" ] } ], @@ -258,14 +269,14 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "PyTorch cuda Inference time = 16.54 ms\n" + "PyTorch cuda Inference time = 16.57 ms\n" ] } ], @@ -297,17 +308,15 @@ "### CUDA and cuDNN Path\n", "onnxruntime-gpu has dependency on [CUDA](https://developer.nvidia.com/cuda-downloads) and [cuDNN](https://developer.nvidia.com/cudnn):\n", "\n", - "* [onnxruntime-gpu v1.2.0](https://github.com/microsoft/onnxruntime/releases/tag/v1.2.0) requires CUDA Runtime 10.1.243 and CUDNN 7.6.5.32.\n", - "* [onnxruntime-gpu v1.0.0](https://github.com/microsoft/onnxruntime/releases/tag/v1.0.0) ~ v1.1.2 requires CUDA Runtime 10.0 and CUDNN 7.6.\n", + "* [onnxruntime-gpu v1.3.0](https://github.com/microsoft/onnxruntime/tree/rel-1.3.0#system-requirements) requires CUDA Runtime 10.1 and CUDNN 7.6.5.\n", + "* [onnxruntime-gpu v1.2.0](https://github.com/microsoft/onnxruntime/releases/tag/v1.2.0) requires CUDA Runtime 10.1 and CUDNN 7.6.5.\n", "\n", - "During installing PyTorch 1.4, we installed cudatoolkit 10.1.243 in this conda environment. That shall be good for onnxruntime-gpu 1.2.0 in Jupyter Notebook.\n", - "\n", - "If you use onnxruntime-gpu 1.0.0 ~ 1.1.2, you will have to install CUDA and CUDNN, then add them to path like the following:" + "During installing PyTorch 1.5, we installed cudatoolkit 10.1.243 in this conda environment. That shall be good for onnxruntime-gpu 1.3.0 in Jupyter Notebook." ] }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 8, "metadata": {}, "outputs": [], "source": [ @@ -316,8 +325,8 @@ "\n", "if add_cuda_path:\n", " # Add path of CUDA 10.0 and CUDNN 7.6 for onnxruntime-gpu 1.0.0 ~ 1.1.2\n", - " cuda_dir = 'D:/NVidia/CUDA/v10.0/bin'\n", - " cudnn_dir = 'D:/NVidia/CUDA/v10.0/bin'\n", + " cuda_dir = 'D:/NVidia/CUDA/v10.1/bin'\n", + " cudnn_dir = 'D:/NVidia/CUDA/v10.1/bin'\n", " if not (os.path.exists(cuda_dir) and os.path.exists(cudnn_dir)):\n", " raise ValueError(\"Please specify correct path for CUDA and cuDNN. Otherwise onnxruntime cannot be imported.\")\n", " else:\n", @@ -342,7 +351,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 9, "metadata": {}, "outputs": [], "source": [ @@ -360,14 +369,14 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "OnnxRuntime gpu Inference time = 3.72 ms\n" + "OnnxRuntime gpu Inference time = 4.43 ms\n" ] } ], @@ -415,7 +424,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 11, "metadata": { "scrolled": true }, @@ -426,9 +435,9 @@ "text": [ "***** Verifying correctness *****\n", "PyTorch and ONNX Runtime output 0 are close: True\n", - "maximum_diff=0.005700558423995972 average_diff=0.0005014391499571502\n", + "maximum_diff=9.499490261077881e-07 average_diff=1.4225952327251434e-07\n", "PyTorch and ONNX Runtime output 1 are close: True\n", - "maximum_diff=0.0023995935916900635 average_diff=0.00045171938836574554\n" + "maximum_diff=6.92903995513916e-07 average_diff=1.2441887520253658e-07\n" ] } ], @@ -454,7 +463,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 12, "metadata": { "scrolled": true }, @@ -490,7 +499,7 @@ " 0, 0, 0, 0, 0, 0, 0, 0]], device='cuda:0')}" ] }, - "execution_count": 11, + "execution_count": 12, "metadata": {}, "output_type": "execute_result" } @@ -509,7 +518,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 13, "metadata": {}, "outputs": [ { @@ -517,7 +526,7 @@ "output_type": "stream", "text": [ "Average length 101\n", - "OnnxRuntime gpu Inference time with actual sequence length = 3.44 ms\n" + "OnnxRuntime gpu Inference time with actual sequence length = 4.23 ms\n" ] } ], @@ -547,12 +556,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Let's compare the output and see whether the results are close." + "Let's compare the output and see whether the results are close.\n", + "\n", + "**Note**: Need end-to-end evaluation on performance and accuracy if you use this strategy." ] }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 14, "metadata": {}, "outputs": [ { @@ -600,12 +611,11 @@ "Example Usage:\n", "```\n", "from onnxruntime_tools import optimizer\n", - "optimized_model = optimizer.optimize_model(export_model_path, model_type='bert', num_heads=12, hidden_size=768, use_gpu=True)\n", - "optimized_model.convert_model_float32_to_float16()\n", + "optimized_model = optimizer.optimize_model(export_model_path, model_type='bert', num_heads=12, hidden_size=768)\n", "optimized_model.save_model_to_file(optimized_model_path)\n", "```\n", "\n", - "You can also use optimizer_cli as the following." + "You can also use optimizer_cli like the following:" ] }, { @@ -627,32 +637,34 @@ "name": "stdout", "output_type": "stream", "text": [ - "optimizer.py: Save optimized model by onnxruntime to ./onnx\\bert-base-cased-squad_opset11_ort_gpu.onnx\n", - "optimizer.py: Use OnnxRuntime to optimize and save the optimized model to ./onnx\\bert-base-cased-squad_opset11_ort_gpu.onnx\n", - " BertOnnxModel.py: Fused LayerNormalization count: 0\n", - " BertOnnxModel.py: Fused Reshape count:0\n", - " BertOnnxModel.py: Fused SkipLayerNormalization count: 24\n", - " BertOnnxModel.py: Fused Attention count:0\n", - " BertOnnxModel.py: skip embed layer fusion since mask input is not found\n", - " BertOnnxModel.py: Fused SkipLayerNormalization with Bias count:24\n", - " BertOnnxModel.py: opset verion: 11\n", - " OnnxModel.py: Output model to ./onnx/bert-base-cased-squad_opt_gpu_fp32.onnx\n", - " BertOnnxModel.py: EmbedLayer=1, Attention=12, Gelu=12, LayerNormalization=24, Succesful=True\n", - "optimizer.py: The output model is fully optimized.\n" + "optimize_by_onnxruntime: Save optimized model by onnxruntime to ./onnx/bert-base-cased-squad_opset11_o1_cpu.onnx\n", + " apply: Fused LayerNormalization count: 25\n", + " apply: Fused Gelu count: 12\n", + " apply: Fused SkipLayerNormalization count: 25\n", + " apply: Fused Attention count: 12\n", + " prune_graph: Graph pruned: 0 inputs, 0 outputs and 5 nodes are removed\n", + " apply: Fused EmbedLayerNormalization(with mask) count: 1\n", + " prune_graph: Graph pruned: 0 inputs, 0 outputs and 12 nodes are removed\n", + " prune_graph: Graph pruned: 0 inputs, 0 outputs and 0 nodes are removed\n", + " apply: Fused BiasGelu count: 12\n", + " apply: Fused SkipLayerNormalization(add bias) count: 24\n", + " optimize: opset verion: 11\n", + " save_model_to_file: Output model to ./onnx/bert-base-cased-squad_opt_gpu_fp32.onnx\n", + "get_fused_operator_statistics: Optimized operators:{'EmbedLayerNormalization': 1, 'Attention': 12, 'Gelu': 0, 'FastGelu': 0, 'BiasGelu': 12, 'LayerNormalization': 0, 'SkipLayerNormalization': 24}\n", + " main: The model has been fully optimized.\n" ] } ], "source": [ "optimized_fp32_model_path = './onnx/bert-base-cased-squad_opt_{}_fp32.onnx'.format('gpu' if use_gpu else 'cpu')\n", "\n", - "!{sys.executable} -m onnxruntime_tools.optimizer_cli --input $export_model_path --output $optimized_fp32_model_path --input_int32" + "!python -m onnxruntime_tools.optimizer_cli --input $export_model_path --output $optimized_fp32_model_path" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Note: you might change \n", "#### Optimized Graph\n", "We can open the optimized model using [Netron](https://github.com/lutzroeder/netron) to visualize.\n", "\n", @@ -671,7 +683,7 @@ "import netron\n", "\n", "# change it to True if want to view the optimized model in browser\n", - "enable_netron = True\n", + "enable_netron = False\n", "if enable_netron:\n", " # If you encounter error \"access a socket in a way forbidden by its access permissions\", install Netron as standalone application instead.\n", " netron.start(optimized_fp32_model_path)" @@ -687,7 +699,7 @@ "\n", "Note that the test uses fixed sequence length. If you use [dynamic sequence length](#Inference-with-Actual-Sequence-Length), actual performance depends on the distribution of sequence length.\n", "\n", - "Note that this tool measures performance of inference using OnnxRuntime Python API. " + "**Attention**: Latency numbers from Jupyter Notebook are not accurate. See [Attional Info](#7.-Additional-Info) for more info." ] }, { @@ -699,16 +711,32 @@ "name": "stdout", "output_type": "stream", "text": [ + "test setting TestSetting(batch_size=1, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=None, omp_wait_policy=None, intra_op_num_threads=None, seed=3, verbose=False, inclusive=False, extra_latency=True)\n", "Generating 1000 samples for batch_size=1 sequence_length=128\n", - "Extra latency for converting inputs to contiguous: 0.01 ms\n", - "Test summary is saved to onnx\\perf_results_GPU_B1_S128_20200313-152329.txt\n" + "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=0,OMP_NUM_THREADS=,OMP_WAIT_POLICY=,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 4.92 ms, Throughput = 203.24 QPS\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 4.90 ms, Throughput = 203.88 QPS\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=1,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 5.07 ms, Throughput = 197.16 QPS\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 4.82 ms, Throughput = 207.33 QPS\n", + "skip duplicated test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "skip duplicated test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=1,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=1,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 4.93 ms, Throughput = 202.92 QPS\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 4.91 ms, Throughput = 203.55 QPS\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp32.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 4.88 ms, Throughput = 204.90 QPS\n", + "Test summary is saved to onnx/perf_results_GPU_B1_S128_20200617-232134.txt\n" ] } ], "source": [ "GPU_OPTION = '--use_gpu' if use_gpu else ''\n", "\n", - "!{sys.executable} -m onnxruntime_tools.transformers.bert_perf_test --model $optimized_fp32_model_path --batch_size 1 --sequence_length 128 --samples 1000 --test_times 1 --inclusive --all $GPU_OPTION" + "!python -m onnxruntime_tools.transformers.bert_perf_test --model $optimized_fp32_model_path --batch_size 1 --sequence_length 128 --samples 1000 --test_times 1 --inclusive --all $GPU_OPTION" ] }, { @@ -727,7 +755,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Float32 model perf results from ./onnx\\perf_results_GPU_B1_S128_20200313-152329.txt\n" + "Float32 model perf results from ./onnx/perf_results_GPU_B1_S128_20200617-232134.txt\n" ] }, { @@ -768,212 +796,107 @@ " \n", " \n", " 0\n", - " 3.43\n", - " 3.42\n", - " 3.44\n", - " 3.46\n", - " 3.47\n", - " 3.58\n", - " 291.59\n", - " 6\n", + " 4.82\n", + " 4.53\n", + " 4.57\n", + " 5.15\n", + " 7.25\n", + " 8.75\n", + " 207.33\n", " 1\n", + " 12\n", " ACTIVE\n", - " False\n", + " None\n", " True\n", " \n", " \n", " 1\n", - " 3.43\n", - " 3.41\n", - " 3.43\n", - " 3.45\n", - " 3.46\n", - " 3.54\n", - " 291.58\n", - " 6\n", - " 1\n", + " 4.88\n", + " 4.54\n", + " 4.58\n", + " 6.47\n", + " 7.13\n", + " 8.68\n", + " 204.90\n", + " 12\n", + " 12\n", " PASSIVE\n", - " False\n", + " None\n", " True\n", " \n", " \n", " 2\n", - " 3.43\n", - " 3.41\n", - " 3.43\n", - " 3.45\n", - " 3.47\n", - " 3.81\n", - " 291.27\n", - " 6\n", - " 6\n", - " ACTIVE\n", - " False\n", + " 4.90\n", + " 4.54\n", + " 4.57\n", + " 6.16\n", + " 7.64\n", + " 8.82\n", + " 203.88\n", + " 1\n", + " 12\n", + " PASSIVE\n", + " None\n", " True\n", " \n", " \n", " 3\n", - " 3.44\n", - " 3.42\n", - " 3.44\n", - " 3.45\n", - " 3.47\n", - " 3.57\n", - " 290.92\n", - " 0\n", - " \n", - " \n", - " False\n", + " 4.91\n", + " 4.55\n", + " 4.59\n", + " 6.70\n", + " 7.43\n", + " 8.78\n", + " 203.55\n", + " 12\n", + " 12\n", + " ACTIVE\n", + " None\n", " True\n", " \n", " \n", " 4\n", - " 3.44\n", - " 3.43\n", - " 3.45\n", - " 3.46\n", - " 3.47\n", - " 3.55\n", - " 290.46\n", - " 1\n", - " 6\n", - " ACTIVE\n", - " True\n", + " 4.92\n", + " 4.57\n", + " 4.60\n", + " 6.50\n", + " 7.82\n", + " 8.90\n", + " 203.24\n", + " 0\n", + " \n", + " \n", + " None\n", " True\n", " \n", " \n", " 5\n", - " 3.44\n", - " 3.43\n", - " 3.45\n", - " 3.46\n", - " 3.48\n", - " 3.94\n", - " 290.31\n", + " 4.93\n", + " 4.55\n", + " 4.59\n", + " 6.66\n", + " 7.57\n", + " 8.80\n", + " 202.92\n", + " 12\n", " 1\n", - " 6\n", " PASSIVE\n", - " False\n", + " None\n", " True\n", " \n", " \n", " 6\n", - " 3.45\n", - " 3.43\n", - " 3.45\n", - " 3.47\n", - " 3.49\n", - " 3.66\n", - " 290.23\n", - " 6\n", + " 5.07\n", + " 4.56\n", + " 4.61\n", + " 7.19\n", + " 8.11\n", + " 9.01\n", + " 197.16\n", + " 12\n", " 1\n", " ACTIVE\n", - " True\n", - " True\n", - " \n", - " \n", - " 7\n", - " 3.45\n", - " 3.43\n", - " 3.44\n", - " 3.46\n", - " 3.48\n", - " 3.56\n", - " 289.90\n", - " 1\n", - " 6\n", - " PASSIVE\n", - " True\n", - " True\n", - " \n", - " \n", - " 8\n", - " 3.45\n", - " 3.43\n", - " 3.45\n", - " 3.47\n", - " 3.49\n", - " 4.07\n", - " 289.74\n", - " 6\n", - " 6\n", - " PASSIVE\n", - " True\n", - " True\n", - " \n", - " \n", - " 9\n", - " 3.46\n", - " 3.45\n", - " 3.46\n", - " 3.48\n", - " 3.50\n", - " 3.58\n", - " 289.17\n", - " 1\n", - " 6\n", - " ACTIVE\n", - " False\n", - " True\n", - " \n", - " \n", - " 10\n", - " 3.46\n", - " 3.45\n", - " 3.46\n", - " 3.48\n", - " 3.50\n", - " 3.68\n", - " 289.01\n", - " 6\n", - " 6\n", - " ACTIVE\n", - " True\n", - " True\n", - " \n", - " \n", - " 11\n", - " 3.46\n", - " 3.44\n", - " 3.45\n", - " 3.47\n", - " 3.49\n", - " 3.84\n", - " 288.98\n", - " 6\n", - " 6\n", - " PASSIVE\n", - " False\n", - " True\n", - " \n", - " \n", - " 12\n", - " 3.48\n", - " 3.46\n", - " 3.48\n", - " 3.49\n", - " 3.51\n", - " 3.67\n", - " 287.47\n", - " 6\n", - " 1\n", - " PASSIVE\n", - " True\n", - " True\n", - " \n", - " \n", - " 13\n", - " 3.53\n", - " 3.52\n", - " 3.53\n", - " 3.55\n", - " 3.58\n", - " 3.80\n", - " 282.93\n", - " 0\n", - " \n", - " \n", - " True\n", + " None\n", " True\n", " \n", " \n", @@ -981,53 +904,32 @@ "" ], "text/plain": [ - " Latency(ms) Latency_P50 Latency_P75 Latency_P90 Latency_P95 \\\n", - "0 3.43 3.42 3.44 3.46 3.47 \n", - "1 3.43 3.41 3.43 3.45 3.46 \n", - "2 3.43 3.41 3.43 3.45 3.47 \n", - "3 3.44 3.42 3.44 3.45 3.47 \n", - "4 3.44 3.43 3.45 3.46 3.47 \n", - "5 3.44 3.43 3.45 3.46 3.48 \n", - "6 3.45 3.43 3.45 3.47 3.49 \n", - "7 3.45 3.43 3.44 3.46 3.48 \n", - "8 3.45 3.43 3.45 3.47 3.49 \n", - "9 3.46 3.45 3.46 3.48 3.50 \n", - "10 3.46 3.45 3.46 3.48 3.50 \n", - "11 3.46 3.44 3.45 3.47 3.49 \n", - "12 3.48 3.46 3.48 3.49 3.51 \n", - "13 3.53 3.52 3.53 3.55 3.58 \n", + " Latency(ms) Latency_P50 Latency_P75 Latency_P90 Latency_P95 \\\n", + "0 4.82 4.53 4.57 5.15 7.25 \n", + "1 4.88 4.54 4.58 6.47 7.13 \n", + "2 4.90 4.54 4.57 6.16 7.64 \n", + "3 4.91 4.55 4.59 6.70 7.43 \n", + "4 4.92 4.57 4.60 6.50 7.82 \n", + "5 4.93 4.55 4.59 6.66 7.57 \n", + "6 5.07 4.56 4.61 7.19 8.11 \n", "\n", - " Latency_P99 Throughput(QPS) intra_op_num_threads OMP_NUM_THREADS \\\n", - "0 3.58 291.59 6 1 \n", - "1 3.54 291.58 6 1 \n", - "2 3.81 291.27 6 6 \n", - "3 3.57 290.92 0 \n", - "4 3.55 290.46 1 6 \n", - "5 3.94 290.31 1 6 \n", - "6 3.66 290.23 6 1 \n", - "7 3.56 289.90 1 6 \n", - "8 4.07 289.74 6 6 \n", - "9 3.58 289.17 1 6 \n", - "10 3.68 289.01 6 6 \n", - "11 3.84 288.98 6 6 \n", - "12 3.67 287.47 6 1 \n", - "13 3.80 282.93 0 \n", + " Latency_P99 Throughput(QPS) intra_op_num_threads OMP_NUM_THREADS \\\n", + "0 8.75 207.33 1 12 \n", + "1 8.68 204.90 12 12 \n", + "2 8.82 203.88 1 12 \n", + "3 8.78 203.55 12 12 \n", + "4 8.90 203.24 0 \n", + "5 8.80 202.92 12 1 \n", + "6 9.01 197.16 12 1 \n", "\n", - " OMP_WAIT_POLICY contiguous warmup \n", - "0 ACTIVE False True \n", - "1 PASSIVE False True \n", - "2 ACTIVE False True \n", - "3 False True \n", - "4 ACTIVE True True \n", - "5 PASSIVE False True \n", - "6 ACTIVE True True \n", - "7 PASSIVE True True \n", - "8 PASSIVE True True \n", - "9 ACTIVE False True \n", - "10 ACTIVE True True \n", - "11 PASSIVE False True \n", - "12 PASSIVE True True \n", - "13 True True " + " OMP_WAIT_POLICY contiguous warmup \n", + "0 ACTIVE None True \n", + "1 PASSIVE None True \n", + "2 PASSIVE None True \n", + "3 ACTIVE None True \n", + "4 None True \n", + "5 PASSIVE None True \n", + "6 ACTIVE None True " ] }, "execution_count": 18, @@ -1036,7 +938,7 @@ } ], "source": [ - "environment import os\n", + "import os\n", "import glob \n", "import pandas\n", "latest_result_file = max(glob.glob(\"./onnx/perf_results_GPU_B1_S128_*.txt\"), key=os.path.getmtime)\n", @@ -1052,7 +954,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "From above result, we can see that latency is very close for different settings. The default setting (intra_op_num_threads=0, OMP_NUM_THREADS and OMP_WAIT_POLICY does not exist) performs almost on par with the best setting. \n", + "From above result, we can see that latency is very close for different settings. The default setting (intra_op_num_threads=0, OMP_NUM_THREADS and OMP_WAIT_POLICY does not exist) performs the best. \n", "\n", "### Model Results Comparison Tool\n", "\n", @@ -1070,14 +972,14 @@ "name": "stdout", "output_type": "stream", "text": [ - "1 out of 100 results not passed for thresholds (rtol=0.01, atol=0.01).\n", - "maximum absolute difference=0.033498868346214294\n", - "maximum relative difference=20.89048957824707\n" + "100% passed for 100 random inputs given thresholds (rtol=0.01, atol=0.01).\r\n", + "maximum absolute difference=1.9222497940063477e-06\r\n", + "maximum relative difference=0.05027933046221733\r\n" ] } ], "source": [ - "!{sys.executable} -m onnxruntime_tools.transformers.compare_bert_results --baseline_model $export_model_path --optimized_model $optimized_fp32_model_path --batch_size 1 --sequence_length 128 --samples 100 --rtol 0.01 --atol 0.01 $GPU_OPTION" + "!python -m onnxruntime_tools.transformers.compare_bert_results --baseline_model $export_model_path --optimized_model $optimized_fp32_model_path --batch_size 1 --sequence_length 128 --samples 100 --rtol 0.01 --atol 0.01 $GPU_OPTION" ] }, { @@ -1100,24 +1002,27 @@ "name": "stdout", "output_type": "stream", "text": [ - "optimizer.py: Save optimized model by onnxruntime to ./onnx\\bert-base-cased-squad_opset11_ort_gpu.onnx\n", - "optimizer.py: Use OnnxRuntime to optimize and save the optimized model to ./onnx\\bert-base-cased-squad_opset11_ort_gpu.onnx\n", - " BertOnnxModel.py: Fused LayerNormalization count: 0\n", - " BertOnnxModel.py: Fused Reshape count:0\n", - " BertOnnxModel.py: Fused SkipLayerNormalization count: 24\n", - " BertOnnxModel.py: Fused Attention count:0\n", - " BertOnnxModel.py: skip embed layer fusion since mask input is not found\n", - " BertOnnxModel.py: Fused SkipLayerNormalization with Bias count:24\n", - " BertOnnxModel.py: opset verion: 11\n", - " OnnxModel.py: Output model to ./onnx/bert-base-cased-squad_opt_gpu_fp16.onnx\n", - " BertOnnxModel.py: EmbedLayer=1, Attention=12, Gelu=12, LayerNormalization=24, Succesful=True\n", - "optimizer.py: The output model is fully optimized.\n" + "optimize_by_onnxruntime: Save optimized model by onnxruntime to ./onnx/bert-base-cased-squad_opset11_o1_cpu.onnx\n", + " apply: Fused LayerNormalization count: 25\n", + " apply: Fused Gelu count: 12\n", + " apply: Fused SkipLayerNormalization count: 25\n", + " apply: Fused Attention count: 12\n", + " prune_graph: Graph pruned: 0 inputs, 0 outputs and 5 nodes are removed\n", + " apply: Fused EmbedLayerNormalization(with mask) count: 1\n", + " prune_graph: Graph pruned: 0 inputs, 0 outputs and 12 nodes are removed\n", + " prune_graph: Graph pruned: 0 inputs, 0 outputs and 0 nodes are removed\n", + " apply: Fused BiasGelu count: 12\n", + " apply: Fused SkipLayerNormalization(add bias) count: 24\n", + " optimize: opset verion: 11\n", + " save_model_to_file: Output model to ./onnx/bert-base-cased-squad_opt_gpu_fp16.onnx\n", + "get_fused_operator_statistics: Optimized operators:{'EmbedLayerNormalization': 1, 'Attention': 12, 'Gelu': 0, 'FastGelu': 0, 'BiasGelu': 12, 'LayerNormalization': 0, 'SkipLayerNormalization': 24}\n", + " main: The model has been fully optimized.\n" ] } ], "source": [ "optimized_fp16_model_path = './onnx/bert-base-cased-squad_opt_{}_fp16.onnx'.format('gpu' if use_gpu else 'cpu')\n", - "!{sys.executable} -m onnxruntime_tools.optimizer_cli --input $export_model_path --output $optimized_fp16_model_path --float16 --input_int32" + "!python -m onnxruntime_tools.optimizer_cli --input $export_model_path --output $optimized_fp16_model_path --float16" ] }, { @@ -1129,15 +1034,31 @@ "name": "stdout", "output_type": "stream", "text": [ + "test setting TestSetting(batch_size=1, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=None, omp_wait_policy=None, intra_op_num_threads=None, seed=3, verbose=False, inclusive=False, extra_latency=True)\n", "Generating 1000 samples for batch_size=1 sequence_length=128\n", - "Extra latency for converting inputs to contiguous: 0.00 ms\n", - "Test summary is saved to onnx\\perf_results_GPU_B1_S128_20200313-152527.txt\n" + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=0,OMP_NUM_THREADS=,OMP_WAIT_POLICY=,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 3.01 ms, Throughput = 331.90 QPS\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 3.12 ms, Throughput = 320.00 QPS\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=1,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 3.02 ms, Throughput = 331.39 QPS\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 3.01 ms, Throughput = 332.53 QPS\n", + "skip duplicated test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "skip duplicated test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=1,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=1,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 3.04 ms, Throughput = 328.67 QPS\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 3.01 ms, Throughput = 331.72 QPS\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=12,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=PASSIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 3.04 ms, Throughput = 329.32 QPS\n", + "Test summary is saved to onnx/perf_results_GPU_B1_S128_20200617-232234.txt\n" ] } ], "source": [ "GPU_OPTION = '--use_gpu' if use_gpu else ''\n", - "!{sys.executable} -m onnxruntime_tools.transformers.bert_perf_test --model $optimized_fp16_model_path --batch_size 1 --sequence_length 128 --samples 1000 --test_times 1 --inclusive --all $GPU_OPTION" + "!python -m onnxruntime_tools.transformers.bert_perf_test --model $optimized_fp16_model_path --batch_size 1 --sequence_length 128 --samples 1000 --test_times 1 --inclusive --all $GPU_OPTION" ] }, { @@ -1149,7 +1070,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Float32 model perf results from ./onnx\\perf_results_GPU_B1_S128_20200313-152527.txt\n" + "Float32 model perf results from ./onnx/perf_results_GPU_B1_S128_20200617-232234.txt\n" ] }, { @@ -1190,212 +1111,107 @@ " \n", " \n", " 0\n", - " 1.84\n", - " 1.83\n", - " 1.83\n", - " 1.85\n", - " 1.87\n", - " 2.00\n", - " 544.94\n", - " 6\n", + " 3.01\n", + " 2.79\n", + " 2.81\n", + " 2.86\n", + " 5.08\n", + " 7.16\n", + " 332.53\n", " 1\n", - " PASSIVE\n", - " False\n", + " 12\n", + " ACTIVE\n", + " None\n", " True\n", " \n", " \n", " 1\n", - " 1.85\n", - " 1.82\n", - " 1.83\n", - " 1.85\n", - " 1.88\n", - " 2.11\n", - " 541.89\n", + " 3.01\n", + " 2.80\n", + " 2.81\n", + " 2.88\n", + " 4.52\n", + " 7.05\n", + " 331.90\n", " 0\n", " \n", " \n", - " False\n", + " None\n", " True\n", " \n", " \n", " 2\n", - " 1.85\n", - " 1.83\n", - " 1.84\n", - " 1.87\n", - " 1.90\n", - " 2.12\n", - " 541.84\n", - " 6\n", - " 1\n", + " 3.01\n", + " 2.78\n", + " 2.80\n", + " 2.92\n", + " 5.01\n", + " 7.02\n", + " 331.72\n", + " 12\n", + " 12\n", " ACTIVE\n", - " False\n", + " None\n", " True\n", " \n", " \n", " 3\n", - " 1.85\n", - " 1.84\n", - " 1.85\n", - " 1.87\n", - " 1.89\n", - " 2.12\n", - " 541.64\n", + " 3.02\n", + " 2.79\n", + " 2.80\n", + " 2.85\n", + " 6.34\n", + " 7.04\n", + " 331.39\n", + " 12\n", " 1\n", - " 6\n", " ACTIVE\n", - " True\n", + " None\n", " True\n", " \n", " \n", " 4\n", - " 1.85\n", - " 1.83\n", - " 1.84\n", - " 1.86\n", - " 1.89\n", - " 2.09\n", - " 541.61\n", - " 0\n", - " \n", - " \n", - " True\n", + " 3.04\n", + " 2.80\n", + " 2.82\n", + " 2.93\n", + " 5.56\n", + " 7.08\n", + " 329.32\n", + " 12\n", + " 12\n", + " PASSIVE\n", + " None\n", " True\n", " \n", " \n", " 5\n", - " 1.85\n", - " 1.83\n", - " 1.84\n", - " 1.86\n", - " 1.90\n", - " 2.07\n", - " 541.00\n", - " 6\n", - " 6\n", - " ACTIVE\n", - " False\n", + " 3.04\n", + " 2.79\n", + " 2.81\n", + " 2.92\n", + " 6.37\n", + " 7.08\n", + " 328.67\n", + " 12\n", + " 1\n", + " PASSIVE\n", + " None\n", " True\n", " \n", " \n", " 6\n", - " 1.85\n", - " 1.84\n", - " 1.84\n", - " 1.86\n", - " 1.89\n", - " 2.18\n", - " 540.81\n", + " 3.12\n", + " 2.79\n", + " 2.82\n", + " 2.96\n", + " 6.66\n", + " 7.20\n", + " 320.00\n", " 1\n", - " 6\n", + " 12\n", " PASSIVE\n", - " True\n", - " True\n", - " \n", - " \n", - " 7\n", - " 1.85\n", - " 1.83\n", - " 1.84\n", - " 1.86\n", - " 1.89\n", - " 2.08\n", - " 540.38\n", - " 6\n", - " 1\n", - " ACTIVE\n", - " True\n", - " True\n", - " \n", - " \n", - " 8\n", - " 1.85\n", - " 1.83\n", - " 1.84\n", - " 1.85\n", - " 1.88\n", - " 2.16\n", - " 540.01\n", - " 1\n", - " 6\n", - " ACTIVE\n", - " False\n", - " True\n", - " \n", - " \n", - " 9\n", - " 1.86\n", - " 1.83\n", - " 1.84\n", - " 1.87\n", - " 1.90\n", - " 2.14\n", - " 538.67\n", - " 1\n", - " 6\n", - " PASSIVE\n", - " False\n", - " True\n", - " \n", - " \n", - " 10\n", - " 1.86\n", - " 1.84\n", - " 1.85\n", - " 1.87\n", - " 1.90\n", - " 2.24\n", - " 538.10\n", - " 6\n", - " 6\n", - " PASSIVE\n", - " True\n", - " True\n", - " \n", - " \n", - " 11\n", - " 1.86\n", - " 1.84\n", - " 1.85\n", - " 1.87\n", - " 1.89\n", - " 2.29\n", - " 537.35\n", - " 6\n", - " 1\n", - " PASSIVE\n", - " True\n", - " True\n", - " \n", - " \n", - " 12\n", - " 1.88\n", - " 1.85\n", - " 1.86\n", - " 1.88\n", - " 1.93\n", - " 2.31\n", - " 533.32\n", - " 6\n", - " 6\n", - " ACTIVE\n", - " True\n", - " True\n", - " \n", - " \n", - " 13\n", - " 1.88\n", - " 1.84\n", - " 1.87\n", - " 1.96\n", - " 2.11\n", - " 2.47\n", - " 530.87\n", - " 6\n", - " 6\n", - " PASSIVE\n", - " False\n", + " None\n", " True\n", " \n", " \n", @@ -1403,53 +1219,32 @@ "" ], "text/plain": [ - " Latency(ms) Latency_P50 Latency_P75 Latency_P90 Latency_P95 \\\n", - "0 1.84 1.83 1.83 1.85 1.87 \n", - "1 1.85 1.82 1.83 1.85 1.88 \n", - "2 1.85 1.83 1.84 1.87 1.90 \n", - "3 1.85 1.84 1.85 1.87 1.89 \n", - "4 1.85 1.83 1.84 1.86 1.89 \n", - "5 1.85 1.83 1.84 1.86 1.90 \n", - "6 1.85 1.84 1.84 1.86 1.89 \n", - "7 1.85 1.83 1.84 1.86 1.89 \n", - "8 1.85 1.83 1.84 1.85 1.88 \n", - "9 1.86 1.83 1.84 1.87 1.90 \n", - "10 1.86 1.84 1.85 1.87 1.90 \n", - "11 1.86 1.84 1.85 1.87 1.89 \n", - "12 1.88 1.85 1.86 1.88 1.93 \n", - "13 1.88 1.84 1.87 1.96 2.11 \n", + " Latency(ms) Latency_P50 Latency_P75 Latency_P90 Latency_P95 \\\n", + "0 3.01 2.79 2.81 2.86 5.08 \n", + "1 3.01 2.80 2.81 2.88 4.52 \n", + "2 3.01 2.78 2.80 2.92 5.01 \n", + "3 3.02 2.79 2.80 2.85 6.34 \n", + "4 3.04 2.80 2.82 2.93 5.56 \n", + "5 3.04 2.79 2.81 2.92 6.37 \n", + "6 3.12 2.79 2.82 2.96 6.66 \n", "\n", - " Latency_P99 Throughput(QPS) intra_op_num_threads OMP_NUM_THREADS \\\n", - "0 2.00 544.94 6 1 \n", - "1 2.11 541.89 0 \n", - "2 2.12 541.84 6 1 \n", - "3 2.12 541.64 1 6 \n", - "4 2.09 541.61 0 \n", - "5 2.07 541.00 6 6 \n", - "6 2.18 540.81 1 6 \n", - "7 2.08 540.38 6 1 \n", - "8 2.16 540.01 1 6 \n", - "9 2.14 538.67 1 6 \n", - "10 2.24 538.10 6 6 \n", - "11 2.29 537.35 6 1 \n", - "12 2.31 533.32 6 6 \n", - "13 2.47 530.87 6 6 \n", + " Latency_P99 Throughput(QPS) intra_op_num_threads OMP_NUM_THREADS \\\n", + "0 7.16 332.53 1 12 \n", + "1 7.05 331.90 0 \n", + "2 7.02 331.72 12 12 \n", + "3 7.04 331.39 12 1 \n", + "4 7.08 329.32 12 12 \n", + "5 7.08 328.67 12 1 \n", + "6 7.20 320.00 1 12 \n", "\n", - " OMP_WAIT_POLICY contiguous warmup \n", - "0 PASSIVE False True \n", - "1 False True \n", - "2 ACTIVE False True \n", - "3 ACTIVE True True \n", - "4 True True \n", - "5 ACTIVE False True \n", - "6 PASSIVE True True \n", - "7 ACTIVE True True \n", - "8 ACTIVE False True \n", - "9 PASSIVE False True \n", - "10 PASSIVE True True \n", - "11 PASSIVE True True \n", - "12 ACTIVE True True \n", - "13 PASSIVE False True " + " OMP_WAIT_POLICY contiguous warmup \n", + "0 ACTIVE None True \n", + "1 None True \n", + "2 ACTIVE None True \n", + "3 ACTIVE None True \n", + "4 PASSIVE None True \n", + "5 PASSIVE None True \n", + "6 PASSIVE None True " ] }, "execution_count": 22, @@ -1490,34 +1285,56 @@ "name": "stdout", "output_type": "stream", "text": [ + "test setting TestSetting(batch_size=32, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n", "Generating 1000 samples for batch_size=32 sequence_length=128\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=32,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 16.17 ms, Throughput = 1979.41 QPS\n", + "test setting TestSetting(batch_size=1, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n", "Generating 1000 samples for batch_size=1 sequence_length=128\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 3.00 ms, Throughput = 333.83 QPS\n", + "test setting TestSetting(batch_size=2, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n", "Generating 1000 samples for batch_size=2 sequence_length=128\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=2,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 3.59 ms, Throughput = 557.32 QPS\n", + "test setting TestSetting(batch_size=64, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n", "Generating 1000 samples for batch_size=64 sequence_length=128\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=64,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 29.26 ms, Throughput = 2187.15 QPS\n", + "test setting TestSetting(batch_size=4, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n", "Generating 1000 samples for batch_size=4 sequence_length=128\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=4,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 4.32 ms, Throughput = 926.92 QPS\n", + "test setting TestSetting(batch_size=8, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n", "Generating 1000 samples for batch_size=8 sequence_length=128\n", + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=8,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 6.32 ms, Throughput = 1266.63 QPS\n", + "test setting TestSetting(batch_size=16, sequence_length=128, test_cases=1000, test_times=1, contiguous=None, use_gpu=True, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n", "Generating 1000 samples for batch_size=16 sequence_length=128\n", - "Test summary is saved to onnx\\perf_results_GPU_B1-2-4-8-16-32-64_S128_20200313-153014.txt\n" + "Running test: model=bert-base-cased-squad_opt_gpu_fp16.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=16,sequence_length=128,test_cases=1000,test_times=1,contiguous=None,use_gpu=True,warmup=True\n", + "Average latency = 9.60 ms, Throughput = 1666.05 QPS\n", + "Test summary is saved to onnx/perf_results_GPU_B1-2-4-8-16-32-64_S128_20200617-232401.txt\n" ] } ], "source": [ "GPU_OPTION = '--use_gpu' if use_gpu else ''\n", - "!{sys.executable} -m onnxruntime_tools.transformers.bert_perf_test --model $optimized_fp16_model_path --batch_size 1 2 4 8 16 32 64 --sequence_length 128 --samples 1000 --test_times 1 --inclusive $GPU_OPTION" + "THREAD_SETTING = '--intra_op_num_threads 1 --omp_num_threads {} --omp_wait_policy ACTIVE'.format(psutil.cpu_count(logical=True))\n", + "!python -m onnxruntime_tools.transformers.bert_perf_test --model $optimized_fp16_model_path --batch_size 1 2 4 8 16 32 64 --sequence_length 128 --samples 1000 --test_times 1 --inclusive $THREAD_SETTING $GPU_OPTION" ] }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 26, "metadata": { - "scrolled": true + "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Float16 model summary from ./onnx\\perf_results_GPU_B1-2-4-8-16-32-64_S128_20200313-153014.txt\n" + "Float16 model summary from ./onnx/perf_results_GPU_B1-2-4-8-16-32-64_S128_20200617-232401.txt\n" ] }, { @@ -1554,79 +1371,79 @@ " \n", " \n", " 0\n", - " 1.84\n", - " 1.82\n", - " 1.84\n", - " 1.85\n", - " 1.87\n", - " 2.09\n", - " 544.95\n", + " 3.00\n", + " 2.79\n", + " 2.81\n", + " 2.86\n", + " 4.37\n", + " 7.08\n", + " 333.83\n", " 1\n", " \n", " \n", - " 3\n", - " 2.19\n", - " 2.18\n", - " 2.19\n", - " 2.20\n", - " 2.22\n", - " 2.40\n", - " 914.56\n", + " 1\n", + " 3.59\n", + " 3.33\n", + " 3.35\n", + " 3.42\n", + " 6.60\n", + " 7.54\n", + " 557.32\n", " 2\n", " \n", " \n", - " 7\n", - " 2.89\n", - " 2.87\n", - " 2.89\n", - " 2.90\n", - " 2.92\n", - " 3.04\n", - " 1383.38\n", + " 2\n", + " 4.32\n", + " 3.98\n", + " 4.01\n", + " 4.64\n", + " 7.23\n", + " 8.11\n", + " 926.92\n", " 4\n", " \n", " \n", - " 11\n", - " 4.79\n", - " 4.76\n", - " 4.78\n", - " 4.80\n", - " 4.84\n", - " 5.19\n", - " 1671.42\n", + " 3\n", + " 6.32\n", + " 5.94\n", + " 5.97\n", + " 7.61\n", + " 8.96\n", + " 10.12\n", + " 1266.63\n", " 8\n", " \n", " \n", - " 12\n", - " 8.16\n", - " 8.15\n", - " 8.17\n", - " 8.20\n", - " 8.22\n", - " 8.35\n", - " 1960.21\n", + " 4\n", + " 9.60\n", + " 9.22\n", + " 9.25\n", + " 11.32\n", + " 12.33\n", + " 13.34\n", + " 1666.05\n", " 16\n", " \n", " \n", - " 15\n", - " 14.69\n", - " 14.66\n", - " 14.70\n", - " 14.75\n", - " 14.80\n", - " 14.97\n", - " 2178.49\n", + " 5\n", + " 16.17\n", + " 15.80\n", + " 15.90\n", + " 17.38\n", + " 18.80\n", + " 19.93\n", + " 1979.41\n", " 32\n", " \n", " \n", - " 18\n", - " 28.43\n", - " 28.42\n", - " 28.50\n", - " 28.61\n", - " 28.68\n", - " 28.78\n", - " 2251.36\n", + " 6\n", + " 29.26\n", + " 28.89\n", + " 29.01\n", + " 30.63\n", + " 32.53\n", + " 33.28\n", + " 2187.15\n", " 64\n", " \n", " \n", @@ -1634,26 +1451,26 @@ "" ], "text/plain": [ - " Latency(ms) Latency_P50 Latency_P75 Latency_P90 Latency_P95 \\\n", - "0 1.84 1.82 1.84 1.85 1.87 \n", - "3 2.19 2.18 2.19 2.20 2.22 \n", - "7 2.89 2.87 2.89 2.90 2.92 \n", - "11 4.79 4.76 4.78 4.80 4.84 \n", - "12 8.16 8.15 8.17 8.20 8.22 \n", - "15 14.69 14.66 14.70 14.75 14.80 \n", - "18 28.43 28.42 28.50 28.61 28.68 \n", + " Latency(ms) Latency_P50 Latency_P75 Latency_P90 Latency_P95 \\\n", + "0 3.00 2.79 2.81 2.86 4.37 \n", + "1 3.59 3.33 3.35 3.42 6.60 \n", + "2 4.32 3.98 4.01 4.64 7.23 \n", + "3 6.32 5.94 5.97 7.61 8.96 \n", + "4 9.60 9.22 9.25 11.32 12.33 \n", + "5 16.17 15.80 15.90 17.38 18.80 \n", + "6 29.26 28.89 29.01 30.63 32.53 \n", "\n", - " Latency_P99 Throughput(QPS) batch_size \n", - "0 2.09 544.95 1 \n", - "3 2.40 914.56 2 \n", - "7 3.04 1383.38 4 \n", - "11 5.19 1671.42 8 \n", - "12 8.35 1960.21 16 \n", - "15 14.97 2178.49 32 \n", - "18 28.78 2251.36 64 " + " Latency_P99 Throughput(QPS) batch_size \n", + "0 7.08 333.83 1 \n", + "1 7.54 557.32 2 \n", + "2 8.11 926.92 4 \n", + "3 10.12 1266.63 8 \n", + "4 13.34 1666.05 16 \n", + "5 19.93 1979.41 32 \n", + "6 33.28 2187.15 64 " ] }, - "execution_count": 24, + "execution_count": 26, "metadata": {}, "output_type": "execute_result" } @@ -1666,12 +1483,7 @@ "result_data = pandas.read_table(latest_result_file, converters={'OMP_NUM_THREADS': str, 'OMP_WAIT_POLICY':str})\n", "print(\"Float16 model summary from\", latest_result_file)\n", "columns_to_remove = ['model', 'graph_optimization_level', 'test_cases', 'test_times', 'use_gpu', 'warmup', 'sequence_length']\n", - "\n", - "# Set it True to see all rows. Here we only see one setting of intra_op_num_threads==0 and no OMP environment variables\n", - "show_all_rows = False\n", - "if not show_all_rows:\n", - " result_data = result_data.loc[result_data['intra_op_num_threads']==0]\n", - " columns_to_remove.extend(['intra_op_num_threads', 'OMP_NUM_THREADS', 'OMP_WAIT_POLICY', 'contiguous'])\n", + "columns_to_remove.extend(['intra_op_num_threads', 'OMP_NUM_THREADS', 'OMP_WAIT_POLICY', 'contiguous'])\n", "result_data.drop(columns_to_remove, axis=1, inplace=True)\n", "result_data" ] @@ -1682,9 +1494,9 @@ "source": [ "## 7. Additional Info\n", "\n", - "Note that running Jupyter Notebook has slight impact on performance result since Jupyter Notebook is using system resources like CPU etc. You can close Jupyter Notebook and other applications, then run the performance test in a console to get more accurate performance numbers.\n", + "Note that running Jupyter Notebook has significant impact on performance result. You can close Jupyter Notebook and other applications, then run the performance test in a console to get more accurate performance numbers.\n", "\n", - "We have a [benchmark script](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/run_benchmark.sh). It is recommended to use it compare inference speed of OnnxRuntime with PyTorch.\n", + "We have a [benchmark script](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/run_benchmark.sh). It is recommended to use it measure inference speed of OnnxRuntime.\n", "\n", "[OnnxRuntime C API](https://github.com/microsoft/onnxruntime/blob/master/docs/C_API.md) could get slightly better performance than python API. If you use C API in inference, you can use OnnxRuntime_Perf_Test.exe built from source to measure performance instead.\n", "\n", @@ -1693,7 +1505,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 27, "metadata": { "scrolled": true }, @@ -1702,42 +1514,47 @@ "name": "stdout", "output_type": "stream", "text": [ - "{\n", - " \"gpu\": {\n", - " \"driver_version\": \"419.69\",\n", - " \"devices\": [\n", - " {\n", - " \"memory_total\": 17048272896,\n", - " \"memory_available\": 12573147136,\n", - " \"name\": \"Tesla V100-PCIE-16GB\"\n", - " }\n", - " ]\n", - " },\n", - " \"cpu\": {\n", - " \"brand\": \"Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz\",\n", - " \"cores\": 6,\n", - " \"logical_cores\": 6,\n", - " \"hz\": \"2.5940 GHz\",\n", - " \"l2_cache\": \"64\",\n", - " \"l3_cache\": \"0 KB\",\n", - " \"processor\": \"Intel64 Family 6 Model 79 Stepping 1, GenuineIntel\"\n", - " },\n", - " \"memory\": {\n", - " \"total\": 120258613248,\n", - " \"available\": 104351649792\n", - " },\n", - " \"python\": \"3.6.10.final.0 (64 bit)\",\n", - " \"os\": \"Windows-10-10.0.14393-SP0\",\n", - " \"onnxruntime\": {\n", - " \"version\": \"1.2.0\",\n", - " \"support_gpu\": true\n", - " },\n", - " \"pytorch\": {\n", - " \"version\": \"1.4.0\",\n", - " \"support_gpu\": true\n", - " },\n", - " \"tensorflow\": null\n", - "}\n" + "{\r\n", + " \"gpu\": {\r\n", + " \"driver_version\": \"440.64.00\",\r\n", + " \"devices\": [\r\n", + " {\r\n", + " \"memory_total\": 16945512448,\r\n", + " \"memory_available\": 14110883840,\r\n", + " \"name\": \"Tesla V100-PCIE-16GB\"\r\n", + " },\r\n", + " {\r\n", + " \"memory_total\": 16945512448,\r\n", + " \"memory_available\": 16932601856,\r\n", + " \"name\": \"Tesla V100-PCIE-16GB\"\r\n", + " }\r\n", + " ]\r\n", + " },\r\n", + " \"cpu\": {\r\n", + " \"brand\": \"Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz\",\r\n", + " \"cores\": 12,\r\n", + " \"logical_cores\": 12,\r\n", + " \"hz\": \"2.5940 GHz\",\r\n", + " \"l2_cache\": \"256 KB\",\r\n", + " \"l3_cache\": \"35840 KB\",\r\n", + " \"processor\": \"x86_64\"\r\n", + " },\r\n", + " \"memory\": {\r\n", + " \"total\": 236645588992,\r\n", + " \"available\": 222567559168\r\n", + " },\r\n", + " \"python\": \"3.7.7.final.0 (64 bit)\",\r\n", + " \"os\": \"Linux-4.15.0-1089-azure-x86_64-with-debian-stretch-sid\",\r\n", + " \"onnxruntime\": {\r\n", + " \"version\": \"1.3.0\",\r\n", + " \"support_gpu\": true\r\n", + " },\r\n", + " \"pytorch\": {\r\n", + " \"version\": \"1.5.0\",\r\n", + " \"support_gpu\": true\r\n", + " },\r\n", + " \"tensorflow\": null\r\n", + "}\r\n" ] } ], @@ -1748,9 +1565,9 @@ ], "metadata": { "kernelspec": { - "display_name": "torch14_gpu", + "display_name": "gpu_env_py37", "language": "python", - "name": "torch14_gpu" + "name": "gpu_env_py37" }, "language_info": { "codemirror_mode": { @@ -1762,7 +1579,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.6.10" + "version": "3.7.7" } }, "nbformat": 4, diff --git a/onnxruntime/python/tools/transformers/notebooks/Tensorflow_Keras_Bert-Squad_OnnxRuntime_CPU.ipynb b/onnxruntime/python/tools/transformers/notebooks/Tensorflow_Keras_Bert-Squad_OnnxRuntime_CPU.ipynb index d9d0979744..ea7b5f7ab8 100644 --- a/onnxruntime/python/tools/transformers/notebooks/Tensorflow_Keras_Bert-Squad_OnnxRuntime_CPU.ipynb +++ b/onnxruntime/python/tools/transformers/notebooks/Tensorflow_Keras_Bert-Squad_OnnxRuntime_CPU.ipynb @@ -34,7 +34,6 @@ "```console\n", "conda create -n cpu_env python=3.6\n", "conda activate cpu_env\n", - "\n", "conda install -c anaconda ipykernel\n", "conda install -c conda-forge ipywidgets\n", "python -m ipykernel install --user --name=cpu_env\n", @@ -54,17 +53,20 @@ "outputs": [], "source": [ "import sys\n", - "\n", - "!{sys.executable} -m pip install --quiet --upgrade tensorflow==2.1.0\n", + " \n", + "!{sys.executable} -m pip install --quiet --upgrade tensorflow==2.2.0\n", "!{sys.executable} -m pip install --quiet --upgrade onnxruntime\n", - "\n", - "# Install keras2onnx from source, since the latest package (1.6.0) does not support bert models from tensorflow 2.1 currently.\n", - "!{sys.executable} -m pip install --quiet git+https://github.com/microsoft/onnxconverter-common\n", - "!{sys.executable} -m pip install --quiet git+https://github.com/onnx/keras-onnx\n", - " \n", - "# Install other packages used in this notebook. \n", - "!{sys.executable} -m pip install --quiet transformers==2.5.1\n", - "!{sys.executable} -m pip install --quiet wget psutil onnx pytz pandas py-cpuinfo py3nvml" + "!{sys.executable} -m pip install --quiet --upgrade onnxruntime-tools\n", + "!{sys.executable} -m pip install --quiet --upgrade keras2onnx\n", + "!{sys.executable} -m pip install --quiet transformers==2.11.0\n", + "!{sys.executable} -m pip install --quiet wget pandas" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's define some constants:" ] }, { @@ -74,56 +76,28 @@ "outputs": [], "source": [ "# Whether allow overwrite existing script or model.\n", - "enable_overwrite = True\n", + "enable_overwrite = False\n", "\n", "# Number of runs to get average latency.\n", - "total_runs = 100" + "total_runs = 100\n", + "\n", + "# Max sequence length for the export model\n", + "max_sequence_length = 512" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "100% [..............................................................................] 15310 / 15310Downloaded bert_perf_test.py\n", - "100% [................................................................................] 9571 / 9571Downloaded bert_test_data.py\n", - "100% [................................................................................] 7272 / 7272Downloaded compare_bert_results.py\n", - "100% [..............................................................................] 44905 / 44905Downloaded BertOnnxModel.py\n", - "100% [..............................................................................] 21565 / 21565Downloaded BertOnnxModelKeras.py\n", - "100% [..............................................................................] 26114 / 26114Downloaded BertOnnxModelTF.py\n", - "100% [..............................................................................] 22773 / 22773Downloaded OnnxModel.py\n", - "100% [................................................................................] 7795 / 7795Downloaded optimizer.py\n" - ] - } - ], - "source": [ - "import os\n", - "import wget\n", - "\n", - "cache_dir = \"./squad\"\n", - "output_dir = \"./output\"\n", - "script_dir = './bert_scripts'\n", - "\n", - "for directory in [cache_dir, output_dir, script_dir]:\n", - " if not os.path.exists(directory):\n", - " os.makedirs(directory)\n", - "\n", - "# Download scripts for BERT optimization.\n", - "url_prfix = \"https://raw.githubusercontent.com/microsoft/onnxruntime/rel-1.3.0/onnxruntime/python/tools/bert/\"\n", - "script_files = ['bert_perf_test.py', 'bert_test_data.py', 'compare_bert_results.py', 'BertOnnxModel.py', 'BertOnnxModelKeras.py', 'BertOnnxModelTF.py', 'Gpt2OnnxModel.py', 'OnnxModel.py', 'optimizer.py']\n", - "\n", - "for filename in script_files:\n", - " target_file = os.path.join(script_dir, filename)\n", - " if enable_overwrite and os.path.exists(target_file):\n", - " os.remove(target_file)\n", - " if not os.path.exists(target_file):\n", - " wget.download(url_prfix + filename, target_file)\n", - " print(\"Downloaded\", filename)" - ] + "outputs": [], + "source": [ + "import os\n", + "cache_dir = './cached_models'\n", + "output_dir = './onnx_models'\n", + "for directory in [cache_dir, output_dir]:\n", + " if not os.path.exists(directory):\n", + " os.makedirs(directory)" + ] }, { "cell_type": "markdown", @@ -136,7 +110,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Start to load fine-tuned model. This step take a few minutes to download the model (1.3 GB) for the first time." + "Start to load fine-tuned model. This step take a few minutes to download the model for the first time." ] }, { @@ -149,11 +123,14 @@ "source": [ "from transformers import (TFBertForQuestionAnswering, BertTokenizer)\n", "\n", - "model_name_or_path = 'bert-large-uncased-whole-word-masking-finetuned-squad'\n", + "#model_name_or_path = 'bert-large-uncased-whole-word-masking-finetuned-squad'\n", + "model_name_or_path = \"bert-base-cased\"\n", "\n", "# Load model and tokenizer\n", "tokenizer = BertTokenizer.from_pretrained(model_name_or_path, do_lower_case=True, cache_dir=cache_dir)\n", - "model = TFBertForQuestionAnswering.from_pretrained(model_name_or_path, cache_dir=cache_dir)" + "model = TFBertForQuestionAnswering.from_pretrained(model_name_or_path, cache_dir=cache_dir)\n", + "# Needed this to export onnx model with multiple inputs with TF 2.2\n", + "model._saved_model_inputs_spec = None" ] }, { @@ -174,15 +151,18 @@ "name": "stdout", "output_type": "stream", "text": [ - "The answer is: a performance - focused inference engine for on ##nx models\n" + "The answer is: [CLS] what is on ##nx run ##time ? [SEP] on ##nx run ##time is a performance - focused in ##ference engine for on ##nx models\n" ] } ], "source": [ + "import tensorflow as tf\n", "import numpy\n", - "question, text = \"What is ONNX Runtime?\", \"ONNX Runtime is a performance-focused inference engine for ONNX models.\"\n", - "inputs = tokenizer.encode_plus(question, text, add_special_tokens=True, return_tensors='tf')\n", "\n", + "question, text = \"What is ONNX Runtime?\", \"ONNX Runtime is a performance-focused inference engine for ONNX models.\"\n", + "# Pad to max length is needed. Otherwise, position embedding might be truncated by constant folding.\n", + "inputs = tokenizer.encode_plus(question, text, add_special_tokens=True, return_tensors='tf',\n", + " max_length=max_sequence_length, pad_to_max_length=True)\n", "start_scores, end_scores = model(inputs)\n", "\n", "num_tokens = len(inputs[\"input_ids\"][0])\n", @@ -199,7 +179,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "Tensorflow Inference time for sequence length 26 = 227.06 ms\n" + "Tensorflow Inference time for sequence length 512 = 94.62 ms\n" ] } ], @@ -218,36 +198,21 @@ "source": [ "## 3. Export model to ONNX using Keras2onnx\n", "\n", - "Now we use Keras2onnx to export the model to ONNX format. It takes about 18 minutes for the large model." + "Now we use Keras2onnx to export the model to ONNX format. It takes about 3 minutes for bert-base, or 18 minutes for bert-large model." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "The node number after optimization: 5257 -> 3836\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Keras2onnx run time = 1052.26 s\n" - ] - } - ], + "outputs": [], "source": [ "import keras2onnx\n", + "from keras2onnx.proto import keras\n", "\n", "output_model_path = os.path.join(output_dir, 'keras_{}.onnx'.format(model_name_or_path))\n", "\n", "if enable_overwrite or not os.path.exists(output_model_path):\n", - " model.predict(inputs)\n", " start = time.time()\n", " onnx_model = keras2onnx.convert_keras(model, model.name)\n", " keras2onnx.save_model(onnx_model, output_model_path)\n", @@ -281,15 +246,8 @@ "import os\n", "import psutil\n", "\n", - "# You may change the settings in this cell according to Performance Test Tool result after running the whole notebook.\n", - "use_openmp = True\n", - "\n", "# ATTENTION: these environment variables must be set before importing onnxruntime.\n", - "if use_openmp:\n", - " os.environ[\"OMP_NUM_THREADS\"] = str(psutil.cpu_count(logical=True))\n", - "else:\n", - " os.environ[\"OMP_NUM_THREADS\"] = '1'\n", - "\n", + "os.environ[\"OMP_NUM_THREADS\"] = str(psutil.cpu_count(logical=True))\n", "os.environ[\"OMP_WAIT_POLICY\"] = 'ACTIVE'" ] }, @@ -309,7 +267,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "ONNX Runtime cpu inference time for sequence length 26 (model not optimized): 170.70 ms\n" + "ONNX Runtime cpu inference time for sequence length 512 (model not optimized): 630.54 ms\n" ] } ], @@ -318,23 +276,17 @@ "import onnxruntime\n", "import numpy\n", "\n", - "# User might use onnxruntime-gpu for CPU inference.\n", - "if use_openmp and 'CUDAExecutionProvider' in onnxruntime.get_available_providers():\n", - " print(\"warning: onnxruntime-gpu is not built with OpenMP. You might try onnxruntime package.\")\n", - " \n", "sess_options = onnxruntime.SessionOptions()\n", "\n", - "# The following settings enables OpenMP, which is required to get best performance for CPU inference of Bert models.\n", - "if use_openmp:\n", - " sess_options.intra_op_num_threads=1\n", - "else:\n", - " sess_options.intra_op_num_threads=psutil.cpu_count(logical=True)\n", + "# intra_op_num_threads=1 can be used to enable OpenMP in OnnxRuntime 1.2.0.\n", + "# For OnnxRuntime 1.3.0 or later, this does not have effect unless you are using onnxruntime-gpu package.\n", + "sess_options.intra_op_num_threads=1\n", "\n", "# Providers is optional. Only needed when you use onnxruntime-gpu for CPU inference.\n", "session = onnxruntime.InferenceSession(output_model_path, sess_options, providers=['CPUExecutionProvider'])\n", "\n", - "# Use contiguous array as input could improve performance.\n", - "inputs_onnx = {k_: numpy.ascontiguousarray(v_.numpy()) for k_, v_ in inputs.items()}\n", + "batch_size = 1\n", + "inputs_onnx = {k_: numpy.repeat(v_, batch_size, axis=0) for k_, v_ in inputs.items()}\n", "\n", "# Warm up with one run.\n", "results = session.run(None, inputs_onnx)\n", @@ -361,8 +313,8 @@ "WARNING:tensorflow:From :2: _EagerTensorBase.cpu (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n", "Instructions for updating:\n", "Use tf.identity instead.\n", - "start_scores are close: True\n", - "end_scores are close: True\n" + "start_scores are close: False\n", + "end_scores are close: False\n" ] } ], @@ -396,38 +348,14 @@ "cell_type": "code", "execution_count": 11, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - " BertOnnxModelTF.py: Fused LayerNormalization count: 49\n", - "BertOnnxModelKeras.py: Fused Gelu count:24\n", - "BertOnnxModelKeras.py: start processing embedding layer...\n", - "BertOnnxModelKeras.py: Found word embedding. name:tf_bert_for_question_answering/bert/embeddings/Gather/resource:0, shape:(30522, 1024)\n", - "BertOnnxModelKeras.py: Found word embedding. name:tf_bert_for_question_answering/bert/embeddings/position_embeddings/embedding_lookup/413066:0, shape:(512, 1024)\n", - "BertOnnxModelKeras.py: Found segment embedding. name:tf_bert_for_question_answering/bert/embeddings/token_type_embeddings/embedding_lookup/413071:0, shape:(2, 1024)\n", - "BertOnnxModelKeras.py: Create Embedding node\n", - " OnnxModel.py: Graph pruned: 0 inputs, 0 outputs and 9 nodes are removed\n", - "BertOnnxModelKeras.py: Fused mask\n", - "BertOnnxModelKeras.py: Skip consequent Reshape count: 24\n", - " BertOnnxModel.py: Fused Reshape count:0\n", - " BertOnnxModel.py: Fused SkipLayerNormalization count: 48\n", - "BertOnnxModelKeras.py: Fused Attention count:24\n", - " BertOnnxModel.py: Fused SkipLayerNormalization with Bias count:24\n", - "BertOnnxModelKeras.py: Remove 96 Reshape nodes.\n", - " OnnxModel.py: Graph pruned: 0 inputs, 0 outputs and 2160 nodes are removed\n", - " BertOnnxModel.py: opset verion: 11\n", - " OnnxModel.py: Output model to ./output\\keras_bert_large_opt_cpu.onnx\n", - " BertOnnxModel.py: EmbedLayer=1, Attention=24, Gelu=24, LayerNormalization=48, Succesful=True\n", - "optimizer.py: The output model is fully optimized.\n" - ] - } - ], + "outputs": [], "source": [ - "optimized_model_path = os.path.join(output_dir, 'keras_bert_large_opt_cpu.onnx')\n", + "optimized_model_path = os.path.join(output_dir, 'keras_{}_opt_cpu.onnx'.format(model_name_or_path))\n", "\n", - "%run bert_scripts/optimizer.py --input $output_model_path --output $optimized_model_path --model_type bert_keras --num_heads 16 --hidden_size 1024" + "from onnxruntime_tools import optimizer\n", + "optimized_model = optimizer.optimize_model(output_model_path, model_type='bert_keras', num_heads=12, hidden_size=768)\n", + "optimized_model.use_dynamic_axes()\n", + "optimized_model.save_model_to_file(optimized_model_path)" ] }, { @@ -446,7 +374,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "ONNX Runtime cpu inference time on optimized model: 133.35 ms\n" + "ONNX Runtime cpu inference time on optimized model: 369.18 ms\n" ] } ], @@ -474,8 +402,8 @@ "output_type": "stream", "text": [ "***** Verifying correctness (before and after optimization) *****\n", - "start_scores are close: True\n", - "end_scores are close: True\n" + "start_scores are close: False\n", + "end_scores are close: False\n" ] } ], @@ -506,14 +434,14 @@ "output_type": "stream", "text": [ "100% passed for 10 random inputs given thresholds (rtol=0.001, atol=0.0001).\n", - "maximum absolute difference=2.3484230041503906e-05\n", - "maximum relative difference=0.00013404049968812615\n" + "maximum absolute difference=1.2461096048355103e-06\n", + "maximum relative difference=0.006510902661830187\n" ] } ], "source": [ - "# The base model is exported using sequence length 26\n", - "%run ./bert_scripts/compare_bert_results.py --baseline_model $output_model_path --optimized_model $optimized_model_path --batch_size 1 --sequence_length 26 --samples 10" + "# The baseline model is exported using max sequence length, and no dynamic axes\n", + "!{sys.executable} -m onnxruntime_tools.transformers.compare_bert_results --baseline_model $output_model_path --optimized_model $optimized_model_path --batch_size 1 --sequence_length $max_sequence_length --samples 10" ] }, { @@ -524,9 +452,7 @@ "\n", "This tool measures performance of BERT model inference using OnnxRuntime Python API.\n", "\n", - "The following command will create 100 samples of batch_size 1 and sequence length 128 to run inference, then calculate performance numbers like average latency and throughput etc. \n", - "\n", - "It takes about 20 minutes to run this test. You can remove --all to reduce number of settings in the test." + "The following command will create 100 samples of batch_size 1 and sequence length 128 to run inference, then calculate performance numbers like average latency and throughput etc." ] }, { @@ -538,14 +464,18 @@ "name": "stdout", "output_type": "stream", "text": [ + "Running test: model=keras_bert-base-cased_opt_cpu.onnx,graph_optimization_level=ENABLE_ALL,intra_op_num_threads=1,OMP_NUM_THREADS=12,OMP_WAIT_POLICY=ACTIVE,batch_size=1,sequence_length=128,test_cases=100,test_times=1,contiguous=None,use_gpu=False,warmup=True\n", + "Average latency = 99.01 ms, Throughput = 10.10 QPS\n", + "test setting TestSetting(batch_size=1, sequence_length=128, test_cases=100, test_times=1, contiguous=None, use_gpu=False, warmup=True, omp_num_threads=12, omp_wait_policy='ACTIVE', intra_op_num_threads=1, seed=3, verbose=False, inclusive=False, extra_latency=True)\n", "Generating 100 samples for batch_size=1 sequence_length=128\n", - "Extra latency for converting inputs to contiguous: 0.04 ms\n", - "Test summary is saved to output\\perf_results_CPU_B1_S128_20200319-141051.txt\n" + "Test summary is saved to onnx_models\\perf_results_CPU_B1_S128_20200617-210258.txt\n" ] } ], "source": [ - "%run ./bert_scripts/bert_perf_test.py --model $optimized_model_path --batch_size 1 --sequence_length 128 --samples 100 --test_times 1 --inclusive --all" + "THREAD_SETTING = '--intra_op_num_threads 1 --omp_num_threads {} --omp_wait_policy ACTIVE'.format(psutil.cpu_count(logical=True))\n", + "\n", + "!{sys.executable} -m onnxruntime_tools.transformers.bert_perf_test --model $optimized_model_path --batch_size 1 --sequence_length 128 --samples 100 --test_times 1 --inclusive $THREAD_SETTING\n" ] }, { @@ -564,8 +494,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "./output\\perf_results_CPU_B1_S128_20200319-141051.txt\n", - "The best setting is: use openmp; NO contiguous array\n" + "./onnx_models\\perf_results_CPU_B1_S128_20200617-210258.txt\n" ] }, { @@ -603,298 +532,22 @@ " 0\n", " 1\n", " 12\n", - " PASSIVE\n", - " False\n", - " 254.20\n", - " 277.38\n", - " 3.93\n", - " \n", - " \n", - " 1\n", - " 1\n", - " 12\n", - " PASSIVE\n", - " True\n", - " 255.47\n", - " 283.56\n", - " 3.91\n", - " \n", - " \n", - " 2\n", - " 1\n", - " 12\n", " ACTIVE\n", - " False\n", - " 274.95\n", - " 334.49\n", - " 3.64\n", - " \n", - " \n", - " 3\n", - " 12\n", - " 1\n", - " ACTIVE\n", - " True\n", - " 278.91\n", - " 294.30\n", - " 3.59\n", - " \n", - " \n", - " 4\n", - " 1\n", - " 12\n", - " ACTIVE\n", - " True\n", - " 280.97\n", - " 351.41\n", - " 3.56\n", - " \n", - " \n", - " 5\n", - " 12\n", - " 1\n", - " ACTIVE\n", - " False\n", - " 281.89\n", - " 296.11\n", - " 3.55\n", - " \n", - " \n", - " 6\n", - " 12\n", - " 1\n", - " PASSIVE\n", - " True\n", - " 282.75\n", - " 313.10\n", - " 3.54\n", - " \n", - " \n", - " 7\n", - " 1\n", - " \n", - " \n", - " False\n", - " 284.61\n", - " 356.98\n", - " 3.51\n", - " \n", - " \n", - " 8\n", - " 1\n", - " \n", - " \n", - " True\n", - " 292.11\n", - " 361.00\n", - " 3.42\n", - " \n", - " \n", - " 9\n", - " 1\n", - " 6\n", - " PASSIVE\n", - " False\n", - " 292.28\n", - " 346.47\n", - " 3.42\n", - " \n", - " \n", - " 10\n", - " 12\n", - " 1\n", - " PASSIVE\n", - " False\n", - " 292.66\n", - " 346.73\n", - " 3.42\n", - " \n", - " \n", - " 11\n", - " 1\n", - " 6\n", - " PASSIVE\n", - " True\n", - " 310.60\n", - " 443.16\n", - " 3.22\n", - " \n", - " \n", - " 12\n", - " 1\n", - " 6\n", - " ACTIVE\n", - " True\n", - " 338.61\n", - " 402.03\n", - " 2.95\n", - " \n", - " \n", - " 13\n", - " 6\n", - " 1\n", - " PASSIVE\n", - " True\n", - " 362.84\n", - " 378.08\n", - " 2.76\n", - " \n", - " \n", - " 14\n", - " 6\n", - " 1\n", - " ACTIVE\n", - " True\n", - " 362.96\n", - " 372.77\n", - " 2.76\n", - " \n", - " \n", - " 15\n", - " 6\n", - " 1\n", - " ACTIVE\n", - " False\n", - " 363.42\n", - " 392.19\n", - " 2.75\n", - " \n", - " \n", - " 16\n", - " 6\n", - " 1\n", - " PASSIVE\n", - " False\n", - " 363.76\n", - " 385.01\n", - " 2.75\n", - " \n", - " \n", - " 17\n", - " 1\n", - " 6\n", - " ACTIVE\n", - " False\n", - " 367.37\n", - " 423.17\n", - " 2.72\n", - " \n", - " \n", - " 18\n", - " 6\n", - " 6\n", - " ACTIVE\n", - " False\n", - " 372.56\n", - " 484.90\n", - " 2.68\n", - " \n", - " \n", - " 19\n", - " 6\n", - " 6\n", - " PASSIVE\n", - " True\n", - " 383.58\n", - " 408.13\n", - " 2.61\n", - " \n", - " \n", - " 20\n", - " 6\n", - " 6\n", - " PASSIVE\n", - " False\n", - " 384.07\n", - " 393.32\n", - " 2.60\n", - " \n", - " \n", - " 21\n", - " 6\n", - " 6\n", - " ACTIVE\n", - " True\n", - " 388.30\n", - " 647.23\n", - " 2.58\n", - " \n", - " \n", - " 22\n", - " 0\n", - " \n", - " \n", - " True\n", - " 423.20\n", - " 465.31\n", - " 2.36\n", - " \n", - " \n", - " 23\n", - " 0\n", - " \n", - " \n", - " False\n", - " 448.80\n", - " 550.75\n", - " 2.23\n", + " None\n", + " 99.01\n", + " 130.11\n", + " 10.1\n", " \n", " \n", "\n", "" ], "text/plain": [ - " intra_op_num_threads OMP_NUM_THREADS OMP_WAIT_POLICY contiguous \\\n", - "0 1 12 PASSIVE False \n", - "1 1 12 PASSIVE True \n", - "2 1 12 ACTIVE False \n", - "3 12 1 ACTIVE True \n", - "4 1 12 ACTIVE True \n", - "5 12 1 ACTIVE False \n", - "6 12 1 PASSIVE True \n", - "7 1 False \n", - "8 1 True \n", - "9 1 6 PASSIVE False \n", - "10 12 1 PASSIVE False \n", - "11 1 6 PASSIVE True \n", - "12 1 6 ACTIVE True \n", - "13 6 1 PASSIVE True \n", - "14 6 1 ACTIVE True \n", - "15 6 1 ACTIVE False \n", - "16 6 1 PASSIVE False \n", - "17 1 6 ACTIVE False \n", - "18 6 6 ACTIVE False \n", - "19 6 6 PASSIVE True \n", - "20 6 6 PASSIVE False \n", - "21 6 6 ACTIVE True \n", - "22 0 True \n", - "23 0 False \n", + " intra_op_num_threads OMP_NUM_THREADS OMP_WAIT_POLICY contiguous \\\n", + "0 1 12 ACTIVE None \n", "\n", - " Latency(ms) Latency_P99 Throughput(QPS) \n", - "0 254.20 277.38 3.93 \n", - "1 255.47 283.56 3.91 \n", - "2 274.95 334.49 3.64 \n", - "3 278.91 294.30 3.59 \n", - "4 280.97 351.41 3.56 \n", - "5 281.89 296.11 3.55 \n", - "6 282.75 313.10 3.54 \n", - "7 284.61 356.98 3.51 \n", - "8 292.11 361.00 3.42 \n", - "9 292.28 346.47 3.42 \n", - "10 292.66 346.73 3.42 \n", - "11 310.60 443.16 3.22 \n", - "12 338.61 402.03 2.95 \n", - "13 362.84 378.08 2.76 \n", - "14 362.96 372.77 2.76 \n", - "15 363.42 392.19 2.75 \n", - "16 363.76 385.01 2.75 \n", - "17 367.37 423.17 2.72 \n", - "18 372.56 484.90 2.68 \n", - "19 383.58 408.13 2.61 \n", - "20 384.07 393.32 2.60 \n", - "21 388.30 647.23 2.58 \n", - "22 423.20 465.31 2.36 \n", - "23 448.80 550.75 2.23 " + " Latency(ms) Latency_P99 Throughput(QPS) \n", + "0 99.01 130.11 10.1 " ] }, "execution_count": 16, @@ -909,7 +562,6 @@ "latest_result_file = max(glob.glob(os.path.join(output_dir, \"perf_results_*.txt\")), key=os.path.getmtime)\n", "result_data = pandas.read_table(latest_result_file, converters={'OMP_NUM_THREADS': str, 'OMP_WAIT_POLICY':str})\n", "print(latest_result_file)\n", - "print(\"The best setting is: {} openmp; {} contiguous array\".format('use' if result_data['intra_op_num_threads'].iloc[0] == 1 else 'NO', 'use' if result_data['contiguous'].iloc[0] else 'NO'))\n", "\n", "result_data.drop(['model', 'graph_optimization_level', 'batch_size', 'sequence_length', 'test_cases', 'test_times', 'use_gpu', 'warmup'], axis=1, inplace=True)\n", "result_data.drop(['Latency_P50', 'Latency_P75', 'Latency_P90', 'Latency_P95'], axis=1, inplace=True)\n", @@ -925,7 +577,9 @@ "source": [ "## 6. Additional Info\n", "\n", - "Note that running Jupyter Notebook has slight impact on performance result since Jupyter Notebook is using system resources like CPU and memory etc. It is recommended to close Jupyter Notebook and other applications, then run the performance test tool in a console to get more accurate performance numbers.\n", + "Note that running Jupyter Notebook has impact on performance result since Jupyter Notebook is using system resources like CPU and memory etc. It is recommended to close Jupyter Notebook and other applications, then run the performance test tool in a console to get more accurate performance numbers.\n", + "\n", + "We have a [benchmark script](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/run_benchmark.sh). It is recommended to use it to measure inference speed of OnnxRuntime.\n", "\n", "[OnnxRuntime C API](https://github.com/microsoft/onnxruntime/blob/master/docs/C_API.md) could get slightly better performance than python API. If you use C API in inference, you can use OnnxRuntime_Perf_Test.exe built from source to measure performance instead.\n", "\n", @@ -944,11 +598,11 @@ "text": [ "{\n", " \"gpu\": {\n", - " \"driver_version\": \"441.22\",\n", + " \"driver_version\": \"442.23\",\n", " \"devices\": [\n", " {\n", " \"memory_total\": 8589934592,\n", - " \"memory_available\": 611880960,\n", + " \"memory_available\": 1643134976,\n", " \"name\": \"GeForce GTX 1070\"\n", " }\n", " ]\n", @@ -964,29 +618,36 @@ " },\n", " \"memory\": {\n", " \"total\": 16971259904,\n", - " \"available\": 6245142528\n", + " \"available\": 3282817024\n", " },\n", " \"python\": \"3.6.10.final.0 (64 bit)\",\n", " \"os\": \"Windows-10-10.0.18362-SP0\",\n", " \"onnxruntime\": {\n", - " \"version\": \"1.2.0\",\n", + " \"version\": \"1.3.0\",\n", " \"support_gpu\": false\n", " },\n", " \"pytorch\": {\n", - " \"version\": \"1.4.0+cpu\",\n", + " \"version\": \"1.5.0+cpu\",\n", " \"support_gpu\": false\n", " },\n", " \"tensorflow\": {\n", - " \"version\": \"2.1.0\",\n", - " \"git_version\": \"v2.1.0-rc2-17-ge5bf8de410\",\n", + " \"version\": \"2.2.0\",\n", + " \"git_version\": \"v2.2.0-rc4-8-g2b96f3662b\",\n", " \"support_gpu\": true\n", " }\n", "}\n" ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2020-06-17 21:03:03.409601: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll\n" + ] } ], "source": [ - "%run ./bert_scripts/MachineInfo.py --silent" + "!{sys.executable} -m onnxruntime_tools.transformers.machine_info --silent" ] } ],