Quantization tool Overview

This tool supports quantization of an onnx model. quantize() takes a model in ModelProto format and returns the quantized model in ModelProto format.

Quantize an ONNX model

import onnx
from quantize import quantize, QuantizationMode

# Load the onnx model
model = onnx.load('path/to/the/model.onnx')
# Quantize
quantized_model = quantize(model, quantization_mode=QuantizationMode.IntegerOps)
# Save the quantized model
onnx.save(quantized_model, 'path/to/the/quantized_model.onnx')

Examples of various quantization modes

QuantizationMode.IntegerOps with static input quantization: Quantize using integer ops. Inputs/activations are quantized using static scale and zero point values which are specified through "quantization_params" option.

quantized_model = quantize(model, quantization_mode=QuantizationMode.IntegerOps,
                           static=True,
                           quantization_params={
                                'input_1': [np.uint8(113), np.float32(0.05)]
                           })

QuantizationMode.IntegerOps with dynamic input quantization: Quantize using integer ops. Inputs/activations are quantized using dynamic scale and zero point values which are computed while running the model.
```
quantized_model = quantize(model, quantization_mode=QuantizationMode.IntegerOps, static=False)
```

QuantizationMode.QLinearOps with static input quantization: Quantize using QLinear ops. Inputs/activations are quantized using static scale and zero point values which are specified through "input_quantization_params" option. Output scale and zero point values have to be specified using "output_quantization_params" option.

quantized_model = quantize(model, quantization_mode=QuantizationMode.QLinearOps,
                           static=True,
                           quantization_params={
                                'input_1': [np.uint8(113), np.float32(0.05)]
                                'output_1': [np.uint8(113), np.float32(0.05)]
                           })

QuantizationMode.QLinearOps with dynamic input quantization: Quantize using QLinear ops. Inputs/activations are quantized using dynamic scale and zero point values which are computed while running the model. Output scale and zero point values have to be specified using "quantization_params" option.

quantized_model = quantize(model, quantization_mode=QuantizationMode.QLinearOps,
                           static=False,
                           quantization_params={
                                'output_1': [np.uint8(113), np.float32(0.05)]
                           })

Options

See below for a description of all the options to quantize():

model: ModelProto to quantize
per_channel: default: False If True, weights of Conv nodes are quantized per output channel. If False, they are quantized per tensor. Refer QLinearConv for more information.
nbits: default: 8 Number of bits to represent quantized data. Currently only nbits=8 is supported.
quantization_mode: default: QuantizationMode.IntegerOps QuantizationMode.IntegerOps: Quantize using integer ops. Only ConvInteger and MatMulInteger ops are supported now. QuantizationMode.QLinearOps: Quantize using QLinear ops. Only QLinearConv and QLinearMatMul ops are supported now.
static: default:False If True, the inputs/activations are quantized using static scale and zero point values specified through quantization_params. If False, the inputs/activations are quantized using dynamic scale and zero point values computed while running the model.
asymmetric_input_types: default: False If True, weights are quantized into signed integers and inputs/activations into unsigned integers. If False, weights and inputs/activations are quantized into unsigned integers.
force_fusions: default: False If True, nodes added for dynamic quantization are fused. If False, no fusion is applied for nodes which are added for dynamic quantization. This optimization is available from opset 11.
quantization_params: default: None Dictionary to specify the zero point and scale values for inputs to and outputs from conv and matmul nodes. Should be specified when static is set to True. The quantization_params should be specified in the following format: { "input_name": [zero_point, scale] }. zero_point should be of type np.uint8 and scale should be of type np.float32. example: { 'resnet_model/Relu_1:0': [np.uint8(0), np.float32(0.019539741799235344)], 'resnet_model/Relu_2:0': [np.uint8(0), np.float32(0.011359662748873234)] }
nodes_to quantize: default: None List of nodes names to quantize. When this list is not None only the nodes in this list are quantized. exmaple: [ 'Cov__224', 'Conv__252' ]

5.5 KiB Raw Blame History

Quantization tool Overview

Quantize an ONNX model

Examples of various quantization modes

Options

5.5 KiB

Raw Blame History