saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-13 18:08:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

update docs/performance: fix typo (#11372)

Browse source 
Here it should be `max` instead of `abs` for symmetric quantization.
This commit is contained in:
Ender 2022-05-17 02:08:32 +08:00 committed by GitHub
parent 00abe64ce8
commit 8533e50b24
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
docs/performance/quantization.md
View file

@ -28,7 +28,7 @@ Quantization in ONNX Runtime refers to 8 bit linear quantization of an ONNX mode
For symmetric quantization:
```
scale = abs(data_range_max, data_range_min) * 2 / (quantization_range_max - quantization_range_min)
scale = max(abs(data_range_max), abs(data_range_min)) * 2 / (quantization_range_max - quantization_range_min)
```
Zero_point represents zero in the quantization space. It is important that the floating point zero value be exactly representable in quantization space. This is because zero padding is used in many CNNs. If it is not possible to represent 0 uniquely after quantization, it will result in accuracy errors.