diff --git a/docs/performance/quantization.md b/docs/performance/quantization.md index e49651b868..6d7fd4a1f7 100644 --- a/docs/performance/quantization.md +++ b/docs/performance/quantization.md @@ -48,18 +48,112 @@ The picture below shows the equivalent representation with the QOperator and QDQ ![Changes to nodes from basic and extended optimizations](../../images/QDQ_Format.png) ## Quantizing an ONNX model -There are two ways of quantizing a model: dynamic and static. -* **Dynamic quantization**: This method calculates the quantization parameters (scale and zero point) for activations dynamically. +ONNX Runtime provides python APIs for converting 32-bit floating point model to an 8-bit integer +model, a.k.a. quantization. These APIs include pre-processing, dynamic/static quantization, and +debugging. -* **Static quantization**: leverages calibration data to calculate the quantization parameters of activations. Our quantization tool supports three calibration methods: MinMax, Entropy and Percentile. Please refer to [`calibrate.py`](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/quantization/calibrate.py) for details. - -### Quantization API +### Pre-processing {: .no_toc} -Quantization has two main APIs, corresponding to the two quantization methods: `quantize_dynamic()` and `quantize_static()`. +Pre-processing is to transform a float32 model to prepare it for quantization. It consists +of the following three optional steps: + +1. Symbolic shape inference. This is best suited for transformer models. +2. Model optimization: This step uses ONNX Runtime native library to rewrite the +computation graph, including merging computation nodes, eliminating redundancies +to improve runtime efficiency. +3. ONNX shape inference. + +The goal of these steps is to improve quantization quality. Our quantization tool +works best when the tensor's shape is known. Both symbolic shape inference and +ONNX shape inference help figure out tensor shapes. Symbolic shape inference works +best with transformer based models, and ONNX shape inference works with other +models. + +Model optimization performs certain operator fusion that makes quantization tool's +job easier. For instance, a Convolution operator followed by BatchNormalization +can be fused into one during the optimization, which can be quantized very efficiently. + +Unfortunately, a known issue in ONNX Runtime is that model optimization can not output +a model size greater than 2GB. So for large models, optimization must be skipped. + +Pre-processing API is in Python module `onnxruntime.quantization.shape_inference`, +function `quant_pre_process()`. See +[`shape_inference.py`](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/quantization/shape_inference.py). +To read about additional options and finer controls available to pre-processing, +run the following command: + +```console +python -m onnxruntime.quantization.shape_inference --help +``` + +Model optimization may also be performed during quantization. However, this is *NOT* recommended, +even though it's the default behavior due to historical reasons. Model optimization during +quantization creates difficulties for debugging quantization caused accuracy losses, which will +be discussed in [later sections](#qdqdebug). So, it is best to perform model optimization during +pre-processing instead of during quantization. + + +### Dynamic Quantization +{: .no_toc} + +There are two ways of quantizing a model: dynamic and static. Dynamic quantization calculates the +quantization parameters (scale and zero point) for activations dynamically. These calculations +increase the cost of inference, while usually achieve higher accuracy comparing to static ones. + +Python API for dynamic quantization is in module `onnxruntime.quantization.quantize`, function `quantize_dynamic()` + +### Static Quantization +{: .no_toc} + +Static quantization method first runs the model using a set of inputs called calibration data. +During these runs, we compute the quantization parameters for each activations. These quantization +parameters are written as constants to the quantized model and used for all inputs. Our +quantization tool supports three calibration methods: MinMax, Entropy and Percentile. Please refer +to +[`calibrate.py`](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/quantization/calibrate.py) +for details. + +Python API for static quantization is in module `onnxruntime.quantization.quantize`, function +`quantize_static()`. +Please refer to [quantize.py](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/quantization/quantize.py) +for details. + +### Quantization Debugging + +Quantization is not a loss-less transformation. It may negatively affect a model's accuracy. A +solution to this problem is to compare the weights and activations tensors of the original +computation graph vs those of the quantized one, identify where they differ most, and avoid +quantizing these tensors, or choose another quantization/calibration method. This is called +quantization debugging. To facilitate this process, we provide Python APIs for matching +weights and activation tensors between a float32 model and its quantized counterpart. + +API for debugging is in module `onnxruntime.quantization.qdq_loss_debug`, which has the following +functions: + +- Function `create_weight_matching()`. It takes a float32 model and its quantized model, and output + a dictionary that matches the corresponding weights between these two models. +- Function `modify_model_output_intermediate_tensors()`. It takes a float32 or quantized model, and + augment it to save all its activations. +- Function `collect_activations()`. It takes a model augmented by + `modify_model_output_intermediate_tensors()`, and an input data reader, runs the augmented model + to collect all the activations. +- Function `create_activation_matching()`. You can imagine that you run + `collect_activations(modify_model_output_intermediate_tensors())` on both the float32 and its + quantized model, to collect two sets of activations. This function takes these two set of + activations, and matches up corresponding ones, so that they can be easily compared by the user. + +In summary, ONNX Runtimes provides Python APIs for matching up corresponding weights and activation +tensors between a float32 model and its quantized counterpart. This allows the user to easily +compare them to locate where are the biggest differences. + +Model optimization during quantization creates difficulties for this debugging process though, +since it may changes the computation graph in a significant way, resulting in a quantized model +that is drastically different from the original. This makes it hard to match up corresponding +tensors from the two models. As a result, we recommend performing model optimization during +pre-processing instead of the quantization process. -Please refer to [quantize.py](https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/python/tools/quantization/quantize.py) for the quantization options for each method. #### Example {: .no_toc } @@ -75,7 +169,7 @@ model_quant = 'path/to/the/model.quant.onnx' quantized_model = quantize_dynamic(model_fp32, model_quant) ``` -- Static quantization: please refer to the [end-to-end examples](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/quantization). +- Static quantization: please refer to the [end-to-end examples](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/quantization/image_classification/cpu). ### Method selection {: .no_toc}