mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-30 20:18:08 +00:00
Improve Transformer Benchmark for FP16 (#3970)
Disable ORT in offline optimization script (ORT could generate some fused ops (like FusedGemm) which cannot be converted to fp16). Remove some models from benchmark until we have optimizations for them.
This commit is contained in:
Tianlei Wu
GitHub
parent
0d11649bb3
commit
523d70f667
3 changed files with 81 additions and 60 deletions
|
|
@ -16,6 +16,15 @@ This tool can be installed using pip as follows:
|
|||
pip install onnxruntime-tools
|
||||
```
|
||||
|
||||
## Export a transformer model to ONNX
|
||||
PyTorch could export model to ONNX. The tf2onnx and keras2onnx tools can be used to convert model that trained by Tensorflow.
|
||||
Huggingface transformers has a [notebook](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb) shows an example of exporting a pretrained model to ONNX.
|
||||
For Keras2onnx, please refer to its [example script](https://github.com/onnx/keras-onnx/blob/master/applications/nightly_build/test_transformers.py).
|
||||
For tf2onnx, please refer to its [BERT tutorial](https://github.com/onnx/tensorflow-onnx/blob/master/tutorials/BertTutorial.ipynb).
|
||||
|
||||
|
||||
## Model Optimizer
|
||||
|
||||
In your python code, you can use it like the following:
|
||||
|
||||
```python
|
||||
|
|
@ -25,29 +34,16 @@ optimized_model.convert_model_float32_to_float16()
|
|||
optimized_model.save_model_to_file("gpt2_fp16.onnx")
|
||||
```
|
||||
|
||||
You can also use command like the following to optimize model:
|
||||
```console
|
||||
python -m onnxruntime_tools.optimizer_cli --input gpt2.onnx --output gpt2_opt.onnx --model_type gpt2
|
||||
```
|
||||
|
||||
If you want to use the latest script, you can get script files from [here](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/). Then run it like the following:
|
||||
```console
|
||||
python optimizer.py --input gpt2.onnx --output gpt2_opt.onnx --model_type gpt2
|
||||
```
|
||||
|
||||
## Export a transformer model to ONNX
|
||||
PyTorch could export model to ONNX. The tf2onnx and keras2onnx tools can be used to convert model that trained by Tensorflow.
|
||||
Huggingface transformers has a [notebook](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb) shows an example of exporting a pretrained model to ONNX.
|
||||
For Keras2onnx, please refere to its [example script](https://github.com/onnx/keras-onnx/blob/master/applications/nightly_build/test_transformers.py).
|
||||
For tf2onnx, please refer to [its BERT tutorial](https://github.com/onnx/tensorflow-onnx/blob/master/tutorials/BertTutorial.ipynb).
|
||||
|
||||
## Model Optimizer
|
||||
|
||||
Example of using the script optimizer.py to optimize a BERT-large model to run in V100 GPU:
|
||||
You can also use command line. Example of optimizing a BERT-large model to use mixed precision (float16):
|
||||
```console
|
||||
python -m onnxruntime_tools.optimizer_cli --input bert_large.onnx --output bert_large_fp16.onnx --num_heads 16 --hidden_size 1024 --float16
|
||||
```
|
||||
|
||||
You can also download the latest script files from [here](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/). Then run it like the following:
|
||||
```console
|
||||
python optimizer.py --input gpt2.onnx --output gpt2_opt.onnx --model_type gpt2
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
See below for description of some options of optimizer.py:
|
||||
|
|
@ -64,6 +60,10 @@ See below for description of some options of optimizer.py:
|
|||
Exported model ususally uses int64 tensor as input. If this flag is specified, int32 tensors will be used as input, and it could avoid un-necessary Cast nodes and get better performance.
|
||||
- **float16**: (*optional*)
|
||||
By default, model uses float32 in computation. If this flag is specified, half-precision float will be used. This option is recommended for NVidia GPU with Tensor Core like V100 and T4. For older GPUs, float32 is likely faster.
|
||||
**use_gpu**: (*optional*)
|
||||
When opt_level > 1, please set this flag for GPU inference.
|
||||
- **opt_level**: (*optional*)
|
||||
Set a proper graph optimization level of OnnxRuntime: 0 - disable all (default), 1 - basic, 2 - extended, 99 - all. If the value is positive, OnnxRuntime will be used to optimize graph first.
|
||||
- **verbose**: (*optional*)
|
||||
Print verbose information when this flag is specified.
|
||||
|
||||
|
|
@ -86,9 +86,11 @@ For GPT2 models, current optimization does not support past state (both inputs a
|
|||
|
||||
## Benchmark
|
||||
|
||||
There is a benchmark script that measure inference performance of OnnxRuntime, PyTorch or PyTorch+TorchScript on pretrained models of Huggingface Transformers.
|
||||
|
||||
The benchmark script requires PyTorch be installed.
|
||||
|
||||
You can run benchmark script to see the inference speed of OnnxRuntime. Here is an example to run benchmark on pretrained model bert-base-cased on GPU.
|
||||
Here is an example to run benchmark on pretrained model bert-base-cased on GPU.
|
||||
|
||||
```console
|
||||
python -m onnxruntime_tools.transformers.benchmark -g -m bert-base-cased -o -v -b 0
|
||||
|
|
@ -96,17 +98,17 @@ python -m onnxruntime_tools.transformers.benchmark -g -m bert-base-cased -o
|
|||
python -m onnxruntime_tools.transformers.benchmark -g -m bert-base-cased -e torch
|
||||
python -m onnxruntime_tools.transformers.benchmark -g -m bert-base-cased -e torchscript
|
||||
```
|
||||
The first command will generate ONNX models (both before and after optimizations), but not run performance tests since we set batch size to 0. The other three commands will run performance test on three engines: OnnxRuntime, PyTorch and PyTorch+TorchScript.
|
||||
The first command will generate ONNX models (both before and after optimizations), but not run performance tests since batch size is 0. The other three commands will run performance test on each of three engines: OnnxRuntime, PyTorch and PyTorch+TorchScript.
|
||||
|
||||
If you remove -o parameter, optimizer is not used in benchmark.
|
||||
If you remove -o parameter, optimizer script is not used in benchmark.
|
||||
|
||||
If your GPU (like V100 or T4) has TensorCore, you can append --fp16 to the above commands to enable mixed precision using float16.
|
||||
|
||||
If you want to benchmark on CPU, you can remove -g option in the commands.
|
||||
|
||||
Note that our current benchmark on GPT2 model has disabled past state from inputs and outputs.
|
||||
Note that our current benchmark on GPT2 and DistilGPT2 models has disabled past state from inputs and outputs.
|
||||
|
||||
By default, ONNX model has only one input (input_ids). You can use -i parameter to test models with more inputs. For example, we can add "-i 3" to command line to test a bert model with 3 inputs (input_ids, token_type_ids and attention_mask). The performance result might be different. This option only supports OnnxRuntime right now.
|
||||
By default, ONNX model has only one input (input_ids). You can use -i parameter to test models with multiple inputs. For example, we can add "-i 3" to command line to test a bert model with 3 inputs (input_ids, token_type_ids and attention_mask). This option only supports OnnxRuntime right now.
|
||||
|
||||
## Model Verification
|
||||
|
||||
|
|
|
|||
|
|
@ -58,15 +58,16 @@ MODELS = {
|
|||
# No past state inputs for GPT models.
|
||||
"gpt2": (["input_ids"], 11, False, "gpt2"), # no past state inputs & outputs
|
||||
"distilgpt2": (["input_ids"], 11, False, "gpt2"), # no past state inputs & outputs
|
||||
"openai-gpt": (["input_ids"], 11, False, "gpt2"), # no past state inputs
|
||||
|
||||
#"openai-gpt": (["input_ids"], 11, False, "gpt2"), # no past state inputs
|
||||
|
||||
# Models uses Einsum, which need opset version 12 and PyTorch 1.5.0 or above.
|
||||
# Currently OnnxRuntime lacks cuda op for Einsum. GPU inference will be very slow.
|
||||
"albert-base-v2": (["input_ids"], 12, False, "bert"),
|
||||
"xlnet-base-cased": (["input_ids"], 12, False, "bert"),
|
||||
#"albert-base-v2": (["input_ids"], 12, False, "bert"),
|
||||
#"xlnet-base-cased": (["input_ids"], 12, False, "bert"),
|
||||
|
||||
# This model is very large. Need use_external_data_format=True to export it.
|
||||
"xlm-mlm-en-2048": (["input_ids"], 11, True, "bert"),
|
||||
#"xlm-mlm-en-2048": (["input_ids"], 11, True, "bert"),
|
||||
}
|
||||
|
||||
cpu_count = psutil.cpu_count(logical=True)
|
||||
|
|
@ -93,9 +94,12 @@ def load_pretrained_model(model_name, config, cache_dir):
|
|||
return AutoModel.from_pretrained(model_name, config=config, cache_dir=cache_dir)
|
||||
|
||||
|
||||
def create_onnxruntime_session(onnx_model_path, use_gpu):
|
||||
def create_onnxruntime_session(onnx_model_path, use_gpu, enable_all_optimization):
|
||||
import onnxruntime
|
||||
sess_options = onnxruntime.SessionOptions()
|
||||
sess_options.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
|
||||
if not enable_all_optimization:
|
||||
sess_options.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_BASIC
|
||||
|
||||
if (not use_gpu) and (version.parse(onnxruntime.__version__) < version.parse('1.3.0')):
|
||||
# Set intra_op_num_threads = 1 to enable OpenMP for onnxruntime 1.2.0 (cpu)
|
||||
|
|
@ -160,9 +164,8 @@ def build_dynamic_axes(example_inputs, outputs_flatten):
|
|||
return dynamic_axes, output_names
|
||||
|
||||
|
||||
def validate_onnx_model(onnx_model_filename, example_inputs, example_outputs_flatten):
|
||||
use_gpu = "_fp16" in onnx_model_filename
|
||||
test_session = create_onnxruntime_session(onnx_model_filename, use_gpu)
|
||||
def validate_onnx_model(onnx_model_filename, example_inputs, example_outputs_flatten, use_gpu):
|
||||
test_session = create_onnxruntime_session(onnx_model_filename, use_gpu, enable_all_optimization=False)
|
||||
if test_session is None:
|
||||
logger.error(f"{onnx_model_filename} is an invalid ONNX model")
|
||||
return False
|
||||
|
|
@ -179,18 +182,20 @@ def validate_onnx_model(onnx_model_filename, example_inputs, example_outputs_fla
|
|||
|
||||
for i in range(len(example_outputs_flatten)):
|
||||
if not numpy.allclose(example_ort_outputs[i], example_outputs_flatten[i].cpu(), rtol=1e-03, atol=1e-03):
|
||||
logger.error(f"Value of output tensor {i} is not close to expected result")
|
||||
abs_diff = numpy.amax(numpy.abs(example_ort_outputs[i] - example_outputs_flatten[i].cpu()))
|
||||
logger.error(f"Output tensor {i} is not close to expected result. Max diff={abs_diff}")
|
||||
return False
|
||||
|
||||
logger.info(f"inference result of onnxruntime is validated on {onnx_model_filename}")
|
||||
return True
|
||||
|
||||
|
||||
optimize_model_statistics = {}
|
||||
model_fusion_statistics = {}
|
||||
|
||||
|
||||
def optimize_onnx_model(onnx_model_filename, model_type, num_attention_heads, hidden_size, fp16, overwrite):
|
||||
optimized_model_filename = onnx_model_filename.replace(".onnx", "_fp16.onnx" if fp16 else "_fp32.onnx")
|
||||
def optimize_onnx_model(onnx_model_filename, model_type, num_attention_heads, hidden_size, use_gpu, fp16, overwrite):
|
||||
suffix = "_fp{}_{}.onnx".format(16 if fp16 else 32, "gpu" if use_gpu else "cpu")
|
||||
optimized_model_filename = onnx_model_filename.replace(".onnx", suffix)
|
||||
if overwrite or not os.path.exists(optimized_model_filename):
|
||||
from optimizer import optimize_model
|
||||
from BertOnnxModel import BertOptimizationOptions
|
||||
|
|
@ -205,16 +210,22 @@ def optimize_onnx_model(onnx_model_filename, model_type, num_attention_heads, hi
|
|||
hidden_size=hidden_size,
|
||||
opt_level=99,
|
||||
optimization_options=optimization_options,
|
||||
use_gpu=use_gpu,
|
||||
only_onnxruntime=True)
|
||||
optimize_model_statistics[onnx_model_filename] = opt_model.get_fused_operator_statistics()
|
||||
model_fusion_statistics[onnx_model_filename] = opt_model.get_fused_operator_statistics()
|
||||
|
||||
# Use script to optimize model.
|
||||
# Use opt_level <= 1 for models to be converted to fp16, because some fused op (like FusedGemm) has only fp32 and no fp16.
|
||||
# It is better to be conservative so we use opt_level=0 here, in case MemcpyFromHost is added to the graph by OnnxRuntime.
|
||||
opt_model = optimize_model(onnx_model_filename,
|
||||
model_type,
|
||||
num_heads=num_attention_heads,
|
||||
hidden_size=hidden_size,
|
||||
opt_level=99)
|
||||
optimize_model_statistics[optimized_model_filename] = opt_model.get_fused_operator_statistics()
|
||||
opt_level=0,
|
||||
optimization_options=optimization_options,
|
||||
use_gpu=use_gpu,
|
||||
only_onnxruntime=False)
|
||||
model_fusion_statistics[optimized_model_filename] = opt_model.get_fused_operator_statistics()
|
||||
|
||||
if fp16:
|
||||
opt_model.convert_model_float32_to_float16()
|
||||
|
|
@ -224,7 +235,7 @@ def optimize_onnx_model(onnx_model_filename, model_type, num_attention_heads, hi
|
|||
return optimized_model_filename
|
||||
|
||||
|
||||
def export_onnx_model(model_name, cache_dir, input_names, fp16, optimize_onnx, validate_onnx, overwrite):
|
||||
def export_onnx_model(model_name, cache_dir, input_names, use_gpu, fp16, optimize_onnx, validate_onnx, overwrite):
|
||||
config = AutoConfig.from_pretrained(model_name, cache_dir=cache_dir)
|
||||
model = load_pretrained_model(model_name, config=config, cache_dir=cache_dir)
|
||||
model.cpu()
|
||||
|
|
@ -262,15 +273,16 @@ def export_onnx_model(model_name, cache_dir, input_names, fp16, optimize_onnx, v
|
|||
|
||||
is_valid_onnx_model = True
|
||||
if validate_onnx:
|
||||
is_valid_onnx_model = validate_onnx_model(onnx_model_filename, example_inputs, example_outputs_flatten)
|
||||
is_valid_onnx_model = validate_onnx_model(onnx_model_filename, example_inputs, example_outputs_flatten, use_gpu)
|
||||
|
||||
if optimize_onnx or fp16:
|
||||
model_type = MODELS[model_name][3]
|
||||
onnx_model_filename = optimize_onnx_model(onnx_model_filename, model_type, config.num_attention_heads,
|
||||
config.hidden_size, fp16, overwrite)
|
||||
config.hidden_size, use_gpu, fp16, overwrite)
|
||||
|
||||
if validate_onnx:
|
||||
is_valid_onnx_model = validate_onnx_model(onnx_model_filename, example_inputs, example_outputs_flatten)
|
||||
is_valid_onnx_model = validate_onnx_model(onnx_model_filename, example_inputs, example_outputs_flatten,
|
||||
use_gpu)
|
||||
|
||||
return onnx_model_filename, is_valid_onnx_model, config.vocab_size, tokenizer.max_model_input_sizes[model_name]
|
||||
|
||||
|
|
@ -315,11 +327,11 @@ def run_onnxruntime(use_gpu, model_names, fp16, batch_sizes, sequence_lengths, r
|
|||
|
||||
with torch.no_grad():
|
||||
onnx_model_file, is_valid_onnx_model, vocab_size, max_sequence_length = export_onnx_model(
|
||||
model_name, cache_dir, input_names, fp16, optimize_onnx, validate_onnx, overwrite)
|
||||
model_name, cache_dir, input_names, use_gpu, fp16, optimize_onnx, validate_onnx, overwrite)
|
||||
if not is_valid_onnx_model:
|
||||
continue
|
||||
|
||||
ort_session = create_onnxruntime_session(onnx_model_file, use_gpu)
|
||||
ort_session = create_onnxruntime_session(onnx_model_file, use_gpu, enable_all_optimization=True)
|
||||
if ort_session is None:
|
||||
continue
|
||||
|
||||
|
|
@ -468,18 +480,18 @@ def output_summary(results, csv_filename, args):
|
|||
logger.info(f"Summary results are saved to csv file: {csv_filename}")
|
||||
|
||||
|
||||
def output_fusion_statistics(optimize_model_statistics, csv_filename):
|
||||
def output_fusion_statistics(model_fusion_statistics, csv_filename):
|
||||
from transformers import __version__ as transformers_version
|
||||
with open(csv_filename, mode="a", newline='') as csv_file:
|
||||
column_names = ["model_filename", "transformers", "torch"] + list(
|
||||
next(iter(optimize_model_statistics.values())).keys())
|
||||
next(iter(model_fusion_statistics.values())).keys())
|
||||
csv_writer = csv.DictWriter(csv_file, fieldnames=column_names)
|
||||
csv_writer.writeheader()
|
||||
for key in optimize_model_statistics.keys():
|
||||
optimize_model_statistics[key]["transformers"] = transformers_version
|
||||
optimize_model_statistics[key]["torch"] = torch.__version__
|
||||
optimize_model_statistics[key]["model_filename"] = key
|
||||
csv_writer.writerow(optimize_model_statistics[key])
|
||||
for key in model_fusion_statistics.keys():
|
||||
model_fusion_statistics[key]["transformers"] = transformers_version
|
||||
model_fusion_statistics[key]["torch"] = torch.__version__
|
||||
model_fusion_statistics[key]["model_filename"] = key
|
||||
csv_writer.writerow(model_fusion_statistics[key])
|
||||
logger.info(f"Fusion statistics is saved to csv file: {csv_filename}")
|
||||
|
||||
|
||||
|
|
@ -612,9 +624,9 @@ def main():
|
|||
logger.error(f"Exception", exc_info=True)
|
||||
|
||||
time_stamp = datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
if optimize_model_statistics:
|
||||
if model_fusion_statistics:
|
||||
csv_filename = args.fusion_csv or f"benchmark_fusion_{time_stamp}.csv"
|
||||
output_fusion_statistics(optimize_model_statistics, csv_filename)
|
||||
output_fusion_statistics(model_fusion_statistics, csv_filename)
|
||||
|
||||
if len(results) == 0:
|
||||
logger.warning("No any result avaiable.")
|
||||
|
|
|
|||
|
|
@ -76,7 +76,7 @@ def optimize_by_onnxruntime(onnx_model_path, use_gpu=False, optimized_model_path
|
|||
|
||||
if optimized_model_path is None:
|
||||
path_prefix = onnx_model_path[:-5] #remove .onnx suffix
|
||||
optimized_model_path = "{}_ort_{}.onnx".format(path_prefix, "gpu" if use_gpu else "cpu")
|
||||
optimized_model_path = "{}_o{}_{}.onnx".format(path_prefix, opt_level, "gpu" if use_gpu else "cpu")
|
||||
|
||||
sess_options.optimized_model_filepath = optimized_model_path
|
||||
|
||||
|
|
@ -174,11 +174,17 @@ def parse_arguments():
|
|||
parser.add_argument('--verbose', required=False, action='store_true')
|
||||
parser.set_defaults(verbose=False)
|
||||
|
||||
parser.add_argument('--use_gpu', required=False, action='store_true', help="use GPU inference")
|
||||
parser.set_defaults(use_gpu=False)
|
||||
|
||||
parser.add_argument('--only_onnxruntime', required=False, action='store_true', help="optimized by onnxruntime only")
|
||||
parser.set_defaults(only_onnxruntime=False)
|
||||
|
||||
parser.add_argument('--opt_level',
|
||||
required=False,
|
||||
type=int,
|
||||
choices=[0, 1, 2, 99],
|
||||
default=99,
|
||||
default=0,
|
||||
help="onnxruntime optimization level. 0 will disable onnxruntime.")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
|
@ -211,14 +217,15 @@ def optimize_model(input,
|
|||
model_type,
|
||||
num_heads,
|
||||
hidden_size,
|
||||
opt_level=99,
|
||||
opt_level=0,
|
||||
optimization_options=None,
|
||||
use_gpu=False,
|
||||
only_onnxruntime=False):
|
||||
(optimizer_class, producer, run_onnxruntime) = MODEL_CLASSES[model_type]
|
||||
|
||||
input_model_path = input
|
||||
if run_onnxruntime and opt_level > 0:
|
||||
input_model_path = optimize_by_onnxruntime(input_model_path, use_gpu=False, opt_level=opt_level)
|
||||
input_model_path = optimize_by_onnxruntime(input_model_path, use_gpu=use_gpu, opt_level=opt_level)
|
||||
|
||||
model = ModelProto()
|
||||
with open(input_model_path, "rb") as f:
|
||||
|
|
@ -254,8 +261,8 @@ def main():
|
|||
|
||||
optimization_options = get_optimization_options(args)
|
||||
|
||||
bert_model = optimize_model(args.input, args.model_type, args.num_heads, args.hidden_size, args.opt_level,
|
||||
optimization_options)
|
||||
bert_model = optimize_model(args.input, args.model_type, args.num_heads, args.hidden_size, opt_level=args.opt_level,
|
||||
optimization_options=optimization_options, use_gpu=args.use_gpu, only_onnxruntime=args.only_onnxruntime)
|
||||
|
||||
if args.float16:
|
||||
bert_model.convert_model_float32_to_float16()
|
||||
|
|
|
|||
Loading…
Reference in a new issue