mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-01 03:45:06 +00:00
Update ORT to 1.5.1 in Bert Quantization Notebook (#5396)
* Update ORT to 1.5.1 in Bert Quantization Notebook
This commit is contained in:
Yufeng Li
GitHub
parent
132ab2230d
commit
b04cf2d229
1 changed files with 109 additions and 101 deletions
|
|
@ -46,12 +46,12 @@
|
|||
"metadata": {},
|
||||
"source": [
|
||||
"### 0.1 Install packages\n",
|
||||
"Let's install nessasary packages to start the tutorial. We will install PyTorch 1.6, OnnxRuntime 1.4.0, latest ONNX, OnnxRuntime-tools, transformers, and sklearn."
|
||||
"Let's install nessasary packages to start the tutorial. We will install PyTorch 1.6, OnnxRuntime 1.5.1, latest ONNX, OnnxRuntime-tools, transformers, and sklearn."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"execution_count": 1,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
|
|
@ -63,56 +63,57 @@
|
|||
"Looking in links: https://download.pytorch.org/whl/torch_stable.html\n",
|
||||
"Requirement already up-to-date: torch==1.6.0+cpu in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (1.6.0+cpu)\n",
|
||||
"Requirement already up-to-date: torchvision==0.7.0+cpu in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (0.7.0+cpu)\n",
|
||||
"Requirement already satisfied, skipping upgrade: future in /home/yufeng/.local/lib/python3.6/site-packages (from torch==1.6.0+cpu) (0.18.2)\n",
|
||||
"Requirement already satisfied, skipping upgrade: numpy in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from torch==1.6.0+cpu) (1.18.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: future in /home/yufeng/.local/lib/python3.6/site-packages (from torch==1.6.0+cpu) (0.18.2)\n",
|
||||
"Requirement already satisfied, skipping upgrade: pillow>=4.1.1 in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from torchvision==0.7.0+cpu) (7.0.0)\n",
|
||||
"Requirement already up-to-date: onnxruntime==1.4.0 in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (1.4.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: protobuf in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from onnxruntime==1.4.0) (3.9.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: numpy>=1.16.6 in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from onnxruntime==1.4.0) (1.18.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: six>=1.9 in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from protobuf->onnxruntime==1.4.0) (1.12.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: setuptools in /home/yufeng/.local/lib/python3.6/site-packages (from protobuf->onnxruntime==1.4.0) (41.0.1)\n",
|
||||
"Looking in indexes: https://test.pypi.org/simple/\n",
|
||||
"Requirement already satisfied: ort-nightly in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (1.4.0.dev202007152)\n",
|
||||
"Requirement already satisfied: numpy>=1.16.6 in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from ort-nightly) (1.18.1)\n",
|
||||
"Requirement already satisfied: protobuf in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from ort-nightly) (3.9.1)\n",
|
||||
"Requirement already satisfied: six>=1.9 in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from protobuf->ort-nightly) (1.12.0)\n",
|
||||
"Requirement already satisfied: setuptools in /home/yufeng/.local/lib/python3.6/site-packages (from protobuf->ort-nightly) (41.0.1)\n",
|
||||
"Requirement already up-to-date: onnxruntime-tools in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (1.4.2)\n",
|
||||
"Requirement already satisfied, skipping upgrade: psutil in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from onnxruntime-tools) (5.6.3)\n",
|
||||
"Collecting onnxruntime==1.5.1\n",
|
||||
" Using cached onnxruntime-1.5.1-cp36-cp36m-manylinux2014_x86_64.whl (3.8 MB)\n",
|
||||
"Requirement already satisfied, skipping upgrade: numpy>=1.16.6 in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from onnxruntime==1.5.1) (1.18.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: protobuf in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from onnxruntime==1.5.1) (3.9.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: six>=1.9 in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from protobuf->onnxruntime==1.5.1) (1.12.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: setuptools in /home/yufeng/.local/lib/python3.6/site-packages (from protobuf->onnxruntime==1.5.1) (41.0.1)\n",
|
||||
"Installing collected packages: onnxruntime\n",
|
||||
" Attempting uninstall: onnxruntime\n",
|
||||
" Found existing installation: onnxruntime 1.4.0\n",
|
||||
" Uninstalling onnxruntime-1.4.0:\n",
|
||||
" Successfully uninstalled onnxruntime-1.4.0\n",
|
||||
"Successfully installed onnxruntime-1.5.1\n",
|
||||
"Requirement already up-to-date: onnxruntime-tools in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (1.5.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: py-cpuinfo in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from onnxruntime-tools) (5.0.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: py3nvml in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from onnxruntime-tools) (0.2.6)\n",
|
||||
"Requirement already satisfied, skipping upgrade: packaging in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from onnxruntime-tools) (19.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: coloredlogs in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from onnxruntime-tools) (14.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: onnx in /home/yufeng/project/onnx (from onnxruntime-tools) (1.6.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: coloredlogs in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from onnxruntime-tools) (14.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: py3nvml in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from onnxruntime-tools) (0.2.6)\n",
|
||||
"Requirement already satisfied, skipping upgrade: psutil in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from onnxruntime-tools) (5.6.3)\n",
|
||||
"Requirement already satisfied, skipping upgrade: numpy in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from onnxruntime-tools) (1.18.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: xmltodict in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from py3nvml->onnxruntime-tools) (0.12.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: attrs in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from packaging->onnxruntime-tools) (19.1.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: six in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from packaging->onnxruntime-tools) (1.12.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: attrs in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from packaging->onnxruntime-tools) (19.1.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from packaging->onnxruntime-tools) (2.4.2)\n",
|
||||
"Requirement already satisfied, skipping upgrade: humanfriendly>=7.1 in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from coloredlogs->onnxruntime-tools) (8.2)\n",
|
||||
"Requirement already satisfied, skipping upgrade: protobuf in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from onnx->onnxruntime-tools) (3.9.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: typing-extensions>=3.6.2.1 in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages/typing_extensions-3.7.4-py3.6.egg (from onnx->onnxruntime-tools) (3.7.4)\n",
|
||||
"Requirement already satisfied, skipping upgrade: humanfriendly>=7.1 in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from coloredlogs->onnxruntime-tools) (8.2)\n",
|
||||
"Requirement already satisfied, skipping upgrade: xmltodict in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from py3nvml->onnxruntime-tools) (0.12.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: setuptools in /home/yufeng/.local/lib/python3.6/site-packages (from protobuf->onnx->onnxruntime-tools) (41.0.1)\n",
|
||||
"Requirement already up-to-date: transformers in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (3.0.2)\n",
|
||||
"Requirement already satisfied, skipping upgrade: regex!=2019.12.17 in /home/yufeng/.local/lib/python3.6/site-packages (from transformers) (2020.4.4)\n",
|
||||
"Requirement already satisfied, skipping upgrade: sentencepiece!=0.1.92 in /home/yufeng/.local/lib/python3.6/site-packages (from transformers) (0.1.85)\n",
|
||||
"Requirement already satisfied, skipping upgrade: numpy in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from transformers) (1.18.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: tokenizers==0.8.1.rc1 in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from transformers) (0.8.1rc1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: dataclasses; python_version < \"3.7\" in /home/yufeng/.local/lib/python3.6/site-packages (from transformers) (0.7)\n",
|
||||
"Requirement already satisfied, skipping upgrade: packaging in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from transformers) (19.1)\n",
|
||||
"Requirement already up-to-date: transformers in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (3.3.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: tokenizers==0.8.1.rc2 in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from transformers) (0.8.1rc2)\n",
|
||||
"Requirement already satisfied, skipping upgrade: tqdm>=4.27 in /home/yufeng/.local/lib/python3.6/site-packages (from transformers) (4.45.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: requests in /home/yufeng/.local/lib/python3.6/site-packages (from transformers) (2.23.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: dataclasses; python_version < \"3.7\" in /home/yufeng/.local/lib/python3.6/site-packages (from transformers) (0.7)\n",
|
||||
"Requirement already satisfied, skipping upgrade: sentencepiece!=0.1.92 in /home/yufeng/.local/lib/python3.6/site-packages (from transformers) (0.1.85)\n",
|
||||
"Requirement already satisfied, skipping upgrade: sacremoses in /home/yufeng/.local/lib/python3.6/site-packages (from transformers) (0.0.38)\n",
|
||||
"Requirement already satisfied, skipping upgrade: regex!=2019.12.17 in /home/yufeng/.local/lib/python3.6/site-packages (from transformers) (2020.4.4)\n",
|
||||
"Requirement already satisfied, skipping upgrade: requests in /home/yufeng/.local/lib/python3.6/site-packages (from transformers) (2.23.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: numpy in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from transformers) (1.18.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: packaging in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from transformers) (19.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: filelock in /home/yufeng/.local/lib/python3.6/site-packages (from transformers) (3.0.12)\n",
|
||||
"Requirement already satisfied, skipping upgrade: six in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from packaging->transformers) (1.12.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: six in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from sacremoses->transformers) (1.12.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: joblib in /home/yufeng/.local/lib/python3.6/site-packages (from sacremoses->transformers) (0.14.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: click in /home/yufeng/.local/lib/python3.6/site-packages (from sacremoses->transformers) (7.1.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /home/yufeng/.local/lib/python3.6/site-packages (from requests->transformers) (3.0.4)\n",
|
||||
"Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /home/yufeng/.local/lib/python3.6/site-packages (from requests->transformers) (2.9)\n",
|
||||
"Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from requests->transformers) (2019.6.16)\n",
|
||||
"Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/yufeng/.local/lib/python3.6/site-packages (from requests->transformers) (1.25.8)\n",
|
||||
"Requirement already satisfied, skipping upgrade: attrs in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from packaging->transformers) (19.1.0)\n",
|
||||
"Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from packaging->transformers) (2.4.2)\n",
|
||||
"Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/yufeng/.local/lib/python3.6/site-packages (from requests->transformers) (1.25.8)\n",
|
||||
"Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /home/yufeng/.local/lib/python3.6/site-packages (from requests->transformers) (2.9)\n",
|
||||
"Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /home/yufeng/.local/lib/python3.6/site-packages (from requests->transformers) (3.0.4)\n",
|
||||
"Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from requests->transformers) (2019.6.16)\n",
|
||||
"Requirement already satisfied, skipping upgrade: click in /home/yufeng/.local/lib/python3.6/site-packages (from sacremoses->transformers) (7.1.1)\n",
|
||||
"Requirement already satisfied, skipping upgrade: joblib in /home/yufeng/.local/lib/python3.6/site-packages (from sacremoses->transformers) (0.14.1)\n",
|
||||
"Requirement already satisfied: onnx in /home/yufeng/project/onnx (1.6.0)\n",
|
||||
"Requirement already satisfied: sklearn in /home/yufeng/.local/lib/python3.6/site-packages (0.0)\n",
|
||||
"Requirement already satisfied: protobuf in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages (from onnx) (3.9.1)\n",
|
||||
|
|
@ -121,18 +122,23 @@
|
|||
"Requirement already satisfied: typing-extensions>=3.6.2.1 in /home/yufeng/anaconda3/envs/onnx/lib/python3.6/site-packages/typing_extensions-3.7.4-py3.6.egg (from onnx) (3.7.4)\n",
|
||||
"Requirement already satisfied: scikit-learn in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from sklearn) (0.23.1)\n",
|
||||
"Requirement already satisfied: setuptools in /home/yufeng/.local/lib/python3.6/site-packages (from protobuf->onnx) (41.0.1)\n",
|
||||
"Requirement already satisfied: scipy>=0.19.1 in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from scikit-learn->sklearn) (1.5.1)\n",
|
||||
"Requirement already satisfied: joblib>=0.11 in /home/yufeng/.local/lib/python3.6/site-packages (from scikit-learn->sklearn) (0.14.1)\n",
|
||||
"Requirement already satisfied: threadpoolctl>=2.0.0 in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from scikit-learn->sklearn) (2.1.0)\n"
|
||||
"Requirement already satisfied: joblib>=0.11 in /home/yufeng/.local/lib/python3.6/site-packages (from scikit-learn->sklearn) (0.14.1)\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Requirement already satisfied: threadpoolctl>=2.0.0 in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from scikit-learn->sklearn) (2.1.0)\r\n",
|
||||
"Requirement already satisfied: scipy>=0.19.1 in /home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages (from scikit-learn->sklearn) (1.5.1)\r\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Install or upgrade PyTorch 1.6.0 and OnnxRuntime 1.4.0 for CPU-only.\n",
|
||||
"# Install or upgrade PyTorch 1.6.0 and OnnxRuntime 1.5.1 for CPU-only.\n",
|
||||
"import sys\n",
|
||||
"!{sys.executable} -m pip install --upgrade torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html\n",
|
||||
"!{sys.executable} -m pip install --upgrade onnxruntime==1.4.0\n",
|
||||
"!{sys.executable} -m pip install -i https://test.pypi.org/simple/ ort-nightly\n",
|
||||
"!{sys.executable} -m pip install --upgrade onnxruntime==1.5.1\n",
|
||||
"!{sys.executable} -m pip install --upgrade onnxruntime-tools\n",
|
||||
"\n",
|
||||
"# Install other packages used in this notebook.\n",
|
||||
|
|
@ -157,7 +163,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"execution_count": 2,
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
|
|
@ -166,34 +172,27 @@
|
|||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--2020-08-17 14:51:59-- https://raw.githubusercontent.com/huggingface/transformers/f98ef14d161d7bcdc9808b5ec399981481411cc1/utils/download_glue_data.py\n",
|
||||
"--2020-10-06 22:44:47-- https://raw.githubusercontent.com/huggingface/transformers/f98ef14d161d7bcdc9808b5ec399981481411cc1/utils/download_glue_data.py\n",
|
||||
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.188.133\n",
|
||||
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.188.133|:443... connected.\n",
|
||||
"HTTP request sent, awaiting response... 200 OK\n",
|
||||
"Length: 8209 (8.0K) [text/plain]\n",
|
||||
"Saving to: ‘download_glue_data.py’\n",
|
||||
"Saving to: ‘download_glue_data.py.2’\n",
|
||||
"\n",
|
||||
"download_glue_data. 100%[===================>] 8.02K --.-KB/s in 0s \n",
|
||||
"\n",
|
||||
"2020-08-17 14:51:59 (21.6 MB/s) - ‘download_glue_data.py’ saved [8209/8209]\n",
|
||||
"2020-10-06 22:44:47 (26.9 MB/s) - ‘download_glue_data.py.2’ saved [8209/8209]\n",
|
||||
"\n",
|
||||
"Processing MRPC...\n",
|
||||
"Local MRPC data not specified, downloading data from https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_train.txt\n",
|
||||
"\tCompleted!\n",
|
||||
"dev_ids.tsv msr_paraphrase_test.txt test.tsv\n",
|
||||
"dev.tsv msr_paraphrase_train.txt train.tsv\n",
|
||||
"cached_dev_bert-base-uncased_128_mrpc msr_paraphrase_test.txt\t train.tsv\n",
|
||||
"dev_ids.tsv\t\t\t msr_paraphrase_train.txt\n",
|
||||
"dev.tsv\t\t\t\t test.tsv\n",
|
||||
" % Total % Received % Xferd Average Speed Time Time Time Current\n",
|
||||
" Dload Upload Total Spent Left Speed\n",
|
||||
"100 386M 100 386M 0 0 31.7M 0 0:00:12 0:00:12 --:--:-- 34.6M\n",
|
||||
"Archive: MPRC.zip\n",
|
||||
" creating: MRPC/\n",
|
||||
" extracting: MRPC/added_tokens.json \n",
|
||||
" inflating: MRPC/tokenizer_config.json \n",
|
||||
" inflating: MRPC/special_tokens_map.json \n",
|
||||
" inflating: MRPC/config.json \n",
|
||||
" inflating: MRPC/training_args.bin \n",
|
||||
" inflating: MRPC/vocab.txt \n",
|
||||
" inflating: MRPC/pytorch_model.bin \n"
|
||||
"100 386M 100 386M 0 0 32.8M 0 0:00:11 0:00:11 --:--:-- 35.3M\n",
|
||||
"Archive: MPRC.zip\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -237,7 +236,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
|
|
@ -246,7 +245,7 @@
|
|||
"text": [
|
||||
" % Total % Received % Xferd Average Speed Time Time Time Current\n",
|
||||
" Dload Upload Total Spent Left Speed\n",
|
||||
"100 386M 100 386M 0 0 100M 0 0:00:03 0:00:03 --:--:-- 100M\n",
|
||||
"100 386M 100 386M 0 0 77.7M 0 0:00:04 0:00:04 --:--:-- 83.1M\n",
|
||||
"Archive: MPRC.zip\n"
|
||||
]
|
||||
}
|
||||
|
|
@ -281,7 +280,7 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
|
|
@ -385,15 +384,15 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Size (MB): 417.72576904296875\n",
|
||||
"Size (MB): 173.0903205871582\n"
|
||||
"Size (MB): 417.72998905181885\n",
|
||||
"Size (MB): 173.0945405960083\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -428,14 +427,15 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Evaluating: 0%| | 1/408 [00:00<00:45, 9.04it/s]"
|
||||
"\r",
|
||||
"Evaluating: 0%| | 0/408 [00:00<?, ?it/s]"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -449,8 +449,8 @@
|
|||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Evaluating: 100%|██████████| 408/408 [00:32<00:00, 12.46it/s]\n",
|
||||
"Evaluating: 0%| | 2/408 [00:00<00:25, 16.06it/s]"
|
||||
"Evaluating: 100%|██████████| 408/408 [00:33<00:00, 12.12it/s]\n",
|
||||
"Evaluating: 0%| | 0/408 [00:00<?, ?it/s]"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -458,7 +458,7 @@
|
|||
"output_type": "stream",
|
||||
"text": [
|
||||
"{'acc': 0.8602941176470589, 'f1': 0.9018932874354562, 'acc_and_f1': 0.8810937025412575}\n",
|
||||
"Evaluate total time (seconds): 32.8\n",
|
||||
"Evaluate total time (seconds): 33.8\n",
|
||||
"Evaluating PyTorch quantization accuracy and performance:\n"
|
||||
]
|
||||
},
|
||||
|
|
@ -466,7 +466,7 @@
|
|||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Evaluating: 100%|██████████| 408/408 [00:25<00:00, 16.00it/s]"
|
||||
"Evaluating: 100%|██████████| 408/408 [00:25<00:00, 15.74it/s]"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -474,7 +474,7 @@
|
|||
"output_type": "stream",
|
||||
"text": [
|
||||
"{'acc': 0.8578431372549019, 'f1': 0.8993055555555555, 'acc_and_f1': 0.8785743464052287}\n",
|
||||
"Evaluate total time (seconds): 25.5\n"
|
||||
"Evaluate total time (seconds): 26.0\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -662,9 +662,20 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"/home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/modeling_bert.py:191: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
|
||||
" position_ids = self.position_ids[:, :seq_length]\n",
|
||||
"/home/yufeng/anaconda3/envs/pytorch/lib/python3.6/site-packages/transformers/modeling_utils.py:1645: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!\n",
|
||||
" input_tensor.shape == tensor_shape for input_tensor in input_tensors\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import onnxruntime\n",
|
||||
"\n",
|
||||
|
|
@ -723,29 +734,27 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"ONNX full precision model size (MB): 417.6655149459839\n",
|
||||
"ONNX quantized model size (MB): 106.49320030212402\n"
|
||||
"ONNX full precision model size (MB): 417.6690320968628\n",
|
||||
"ONNX quantized model size (MB): 106.49767780303955\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"def quantize_onnx_model(onnx_model_path, quantized_model_path):\n",
|
||||
" from onnxruntime.quantization import quantize, QuantizationMode\n",
|
||||
" from onnxruntime.quantization import quantize_dynamic, QuantType\n",
|
||||
" import onnx\n",
|
||||
" onnx_opt_model = onnx.load(onnx_model_path)\n",
|
||||
" quantized_onnx_model = quantize(onnx_opt_model,\n",
|
||||
" quantization_mode=QuantizationMode.IntegerOps,\n",
|
||||
" symmetric_weight=True,\n",
|
||||
" force_fusions=True)\n",
|
||||
" quantize_dynamic(onnx_model_path,\n",
|
||||
" quantized_model_path,\n",
|
||||
" weight_type=QuantType.QInt8)\n",
|
||||
"\n",
|
||||
" onnx.save(quantized_onnx_model, quantized_model_path)\n",
|
||||
" logger.info(f\"quantized model saved to:{quantized_model_path}\")\n",
|
||||
"\n",
|
||||
"quantize_onnx_model('bert.opt.onnx', 'bert.opt.quant.onnx')\n",
|
||||
|
|
@ -765,15 +774,14 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\r",
|
||||
"Evaluating: 0%| | 0/408 [00:00<?, ?it/s]"
|
||||
"Evaluating: 0%| | 2/408 [00:00<00:33, 11.96it/s]"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -787,8 +795,8 @@
|
|||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Evaluating: 100%|██████████| 408/408 [00:25<00:00, 16.27it/s]\n",
|
||||
"Evaluating: 0%| | 2/408 [00:00<00:20, 19.62it/s]"
|
||||
"Evaluating: 100%|██████████| 408/408 [00:21<00:00, 18.74it/s]\n",
|
||||
"Evaluating: 1%| | 3/408 [00:00<00:19, 20.46it/s]"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -796,7 +804,7 @@
|
|||
"output_type": "stream",
|
||||
"text": [
|
||||
"{'acc': 0.8602941176470589, 'f1': 0.9018932874354562, 'acc_and_f1': 0.8810937025412575}\n",
|
||||
"Evaluate total time (seconds): 25.6\n",
|
||||
"Evaluate total time (seconds): 22.5\n",
|
||||
"Evaluating ONNXRuntime quantization accuracy and performance:\n"
|
||||
]
|
||||
},
|
||||
|
|
@ -804,7 +812,7 @@
|
|||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Evaluating: 100%|██████████| 408/408 [00:19<00:00, 21.10it/s]\n"
|
||||
"Evaluating: 100%|██████████| 408/408 [00:19<00:00, 21.07it/s]"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
|
@ -814,6 +822,13 @@
|
|||
"{'acc': 0.8578431372549019, 'f1': 0.902027027027027, 'acc_and_f1': 0.8799350821409644}\n",
|
||||
"Evaluate total time (seconds): 19.5\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
|
|
@ -916,12 +931,12 @@
|
|||
"| Engine | Full Precision(MB) | Quantized(MB) |\n",
|
||||
"| --- | --- | --- |\n",
|
||||
"| PyTorch 1.6 | 417.7 | 173.1 |\n",
|
||||
"| ORT 1.4 | 417.7 | 106.5 |\n",
|
||||
"| ORT 1.5 | 417.7 | 106.5 |\n",
|
||||
"\n",
|
||||
"### Accuracy\n",
|
||||
"OnnxRuntime achieves a little bit better accuracy and F1 score, even though it has small model size.\n",
|
||||
"\n",
|
||||
"| Metrics | Full Precision | PyTorch 1.6 Quantization | ORT 1.4 Quantization |\n",
|
||||
"| Metrics | Full Precision | PyTorch 1.6 Quantization | ORT 1.5 Quantization |\n",
|
||||
"| --- | --- | --- | --- |\n",
|
||||
"| Accuracy | 0.86029 | 0.85784 | 0.85784 |\n",
|
||||
"| F1 score | 0.90189 | 0.89931 | 0.90203 |\n",
|
||||
|
|
@ -929,21 +944,14 @@
|
|||
"\n",
|
||||
"### Performance\n",
|
||||
"\n",
|
||||
"The evaluation data set has 408 sample. Table below shows the performance on my machine with Intel(R) Xeon(R) E5-1650 v4@3.60GHz CPU. Comparing with full precision, PyTorch quantization achieves ~1.33x speedup, and ORT quantization achieves ~1.34x speedup. And ORT quantization can achieve ~1.27x speedup, comparing with PyTorch quantization. \n",
|
||||
"The evaluation data set has 408 sample. Table below shows the performance on my machine with Intel(R) Xeon(R) E5-1650 v4@3.60GHz CPU. Comparing with PyTorch full precision, PyTorch quantization achieves ~1.33x speedup, and ORT quantization achieves ~1.73x speedup. And ORT quantization can achieve ~1.33x speedup, comparing with PyTorch quantization. \n",
|
||||
"You can run the [benchmark.py](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/transformers/benchmark.py) for comparison on more models.\n",
|
||||
"\n",
|
||||
"|Engine | Full Precision Latency(s) | Quantized(s) |\n",
|
||||
"| --- | --- | --- |\n",
|
||||
"| PyTorch 1.6 | 32.8 | 25.5 |\n",
|
||||
"| ORT 1.4 | 25.6 | 19.5 |"
|
||||
"| PyTorch 1.6 | 33.8 | 22.5 |\n",
|
||||
"| ORT 1.5 | 26.0 | 19.5 |"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
|
|
|
|||
Loading…
Reference in a new issue