mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-24 19:43:35 +00:00
f644ff9fc0
### Description
- Adds general support for per-channel quantized weights to QNN EP (HTP
backend).
- Add QNN EP unit tests for per-channel Conv
- Update quantization tool to allow selecting which ops are quantized
per-channel (and which axis) via tensor-level overrides. Currently,
setting `per_channel=True` assumes all Convs, MatMuls, Gemms,
InstanceNormalization, and LayerNormalization ops should be quantized
per-channel using some assumed default axis.
#### Creating QDQ per-channel Conv model example
```python
from onnxruntime.quantization import CalibrationDataReader, QuantType, quantize
from onnxruntime.quantization.execution_providers.qnn import get_qnn_qdq_config, qnn_preprocess_model
class DataReader(CalibrationDataReader):
# TODO: See ONNX Runtime QNN docs for example of a data reader
# https://onnxruntime.ai/docs/execution-providers/QNN-ExecutionProvider.html#generating-a-quantized-model-x64
pass
if __name__ == "__main__":
input_model_path = "model.onnx"
my_data_reader = DataReader(model_to_quantize)
# Pre-process the original float32 model.
preproc_model_path = "model.preproc.onnx"
model_changed = qnn_preprocess_model(input_model_path, preproc_model_path)
model_to_quantize = preproc_model_path if model_changed else input_model_path
# RELEVANT TO THIS PR:
# Make sure Conv's weight input is quantized to int8/symmetric/per-channel with axis == 0.
# The presence of the 'axis' key indicates that this is a per-channel quantized weight.
init_overrides = {'weight': [{'axis': 0, 'quant_type': QuantType.QInt8, 'symmetric': True}]}
qnn_config = get_qnn_qdq_config(model_to_quantize,
my_data_reader,
init_overrides=init_overrides,
activation_type=QuantType.QUInt16, # uint16 activations
weight_type=QuantType.QUInt8) # uint8 weights by default
quantize(model_to_quantize, "model.qdq.onnx", qnn_config)
```
float32 model:
<img width="683" alt="image"
src="https://github.com/microsoft/onnxruntime/assets/19691973/ca650e49-1ad0-47d8-8c46-17fbc224ca39">
QDQ model (per-channel Conv weight):
<img width="748" alt="image"
src="https://github.com/microsoft/onnxruntime/assets/19691973/6bd469f2-968b-4d11-9526-09b3e71f98e7">
### Motivation and Context
Support more models, especially models with int4 quantized weights.
|
||
|---|---|---|
| .. | ||
| backend | ||
| datasets | ||
| providers/tvm | ||
| tools | ||
| torch_cpp_extensions | ||
| training | ||
| __init__.py | ||
| _ld_preload.py | ||
| _pybind_state.py.in | ||
| exported_symbols.lst | ||
| numpy_helper.h | ||
| onnxruntime_collect_build_info.py | ||
| onnxruntime_inference_collection.py | ||
| onnxruntime_pybind.h | ||
| onnxruntime_pybind_exceptions.cc | ||
| onnxruntime_pybind_exceptions.h | ||
| onnxruntime_pybind_iobinding.cc | ||
| onnxruntime_pybind_mlvalue.cc | ||
| onnxruntime_pybind_mlvalue.h | ||
| onnxruntime_pybind_module.cc | ||
| onnxruntime_pybind_ortvalue.cc | ||
| onnxruntime_pybind_quant.cc | ||
| onnxruntime_pybind_schema.cc | ||
| onnxruntime_pybind_sparse_tensor.cc | ||
| onnxruntime_pybind_state.cc | ||
| onnxruntime_pybind_state.h | ||
| onnxruntime_pybind_state_common.cc | ||
| onnxruntime_pybind_state_common.h | ||
| onnxruntime_validation.py | ||
| pybind.def | ||
| version_script.lds | ||
| version_script_expose_onnx_protobuf.lds | ||