mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-11 17:48:34 +00:00
df28c7d73b
### Description - Uses our own quantization functions instead of the ONNX reference implementation of QuantizeLinear when quantizing weights to int4. - Uses a custom function that packs bytes into 4-bit elements. ### Motivation and Context Running the quantization tool to create QDQ models with int4 weights could take up to 7x longer. This PR uses our own quantization and byte packing utilities to improve performance. #### Measurements Model with ~5M parameters to quantize to int4. - Current implementation: **84.5s** - Only replace ONNX QuantizeLinear implementation: **50.3s** (1.68x speedup) - This PR (replace onnx Q impl, custom packing func): **13.5s** (6.26x speedup) --------- Signed-off-by: adrianlizarraga <adlizarraga@microsoft.com> |
||
|---|---|---|
| .. | ||
| backend | ||
| datasets | ||
| providers/tvm | ||
| tools | ||
| torch_cpp_extensions | ||
| training | ||
| __init__.py | ||
| _ld_preload.py | ||
| _pybind_state.py.in | ||
| exported_symbols.lst | ||
| numpy_helper.h | ||
| onnxruntime_collect_build_info.py | ||
| onnxruntime_inference_collection.py | ||
| onnxruntime_pybind.h | ||
| onnxruntime_pybind_exceptions.cc | ||
| onnxruntime_pybind_exceptions.h | ||
| onnxruntime_pybind_iobinding.cc | ||
| onnxruntime_pybind_mlvalue.cc | ||
| onnxruntime_pybind_mlvalue.h | ||
| onnxruntime_pybind_module.cc | ||
| onnxruntime_pybind_ortvalue.cc | ||
| onnxruntime_pybind_quant.cc | ||
| onnxruntime_pybind_schema.cc | ||
| onnxruntime_pybind_sparse_tensor.cc | ||
| onnxruntime_pybind_state.cc | ||
| onnxruntime_pybind_state.h | ||
| onnxruntime_pybind_state_common.cc | ||
| onnxruntime_pybind_state_common.h | ||
| onnxruntime_validation.py | ||
| pybind.def | ||
| version_script.lds | ||
| version_script_expose_onnx_protobuf.lds | ||