saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-25 19:48:11 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

fix quantization errors of ConvTranspose with per_channel=True (#19996)

Browse source 
### Description
<!-- Describe your changes. -->
 - update axis value for per_channel quantization of QDQConv
   - we should use `axis=1` for ConvTranspose operator.


### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
- this PR fixes https://github.com/microsoft/onnxruntime/issues/19694,
which I have opened
This commit is contained in:
mo-ja 2024-03-30 13:36:15 +09:00 committed by GitHub
parent f3a864217f
commit 00244ea143
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 2 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
onnxruntime/python/tools/quantization/operators/conv.py
View file

@ -247,7 +247,8 @@ class QDQConv(QDQOperatorBase):
self.quantizer.quantize_activation_tensor(node.output[0])
if self.quantizer.is_per_channel():
self.quantizer.quantize_weight_tensor_per_channel(node.input[1], 0)
axis = 0 if node.op_type == "Conv" else 1
self.quantizer.quantize_weight_tensor_per_channel(node.input[1], axis)
else:
self.quantizer.quantize_weight_tensor(node.input[1])