mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-07 17:15:29 +00:00
[DML] MatrixMultiplyIntegerToFloat (#19608)
### Description DML Implementation for [com.microsoft.MatMulIntegerToFloat](https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#com.microsoft.MatMulIntegerToFloat) ``` .\onnxruntime_test_all.exe --gtest_filter="*MatMulIntegerToFloat.*" Note: Google Test filter = *MatMulIntegerToFloat.* [==========] Running 22 tests from 1 test suite. [----------] Global test environment set-up. [----------] 22 tests from MatMulIntegerToFloat [ RUN ] MatMulIntegerToFloat.HasZeroPoint_NoBias_test_S8S8 [ OK ] MatMulIntegerToFloat.HasZeroPoint_NoBias_test_S8S8 (620 ms) [ RUN ] MatMulIntegerToFloat.NoZeroPoint_HasBias_test_S8S8 [ OK ] MatMulIntegerToFloat.NoZeroPoint_HasBias_test_S8S8 (497 ms) [ RUN ] MatMulIntegerToFloat.NoZeroPoint_NoBias_test_S8S8 [ OK ] MatMulIntegerToFloat.NoZeroPoint_NoBias_test_S8S8 (488 ms) [ RUN ] MatMulIntegerToFloat.HasZeroPoint_HasBias_test_S8S8 [ OK ] MatMulIntegerToFloat.HasZeroPoint_HasBias_test_S8S8 (503 ms) [ RUN ] MatMulIntegerToFloat.HasZeroPoint_NoBias_test_U8U8 [ OK ] MatMulIntegerToFloat.HasZeroPoint_NoBias_test_U8U8 (495 ms) [ RUN ] MatMulIntegerToFloat.NoZeroPoint_HasBias_test_U8U8 [ OK ] MatMulIntegerToFloat.NoZeroPoint_HasBias_test_U8U8 (488 ms) [ RUN ] MatMulIntegerToFloat.NoZeroPoint_NoBias_test_U8U8 [ OK ] MatMulIntegerToFloat.NoZeroPoint_NoBias_test_U8U8 (492 ms) [ RUN ] MatMulIntegerToFloat.HasZeroPoint_HasBias_test_U8X8 [ OK ] MatMulIntegerToFloat.HasZeroPoint_HasBias_test_U8X8 (502 ms) [ RUN ] MatMulIntegerToFloat.HasZeroPoint_NoBias_test_S8U8 [ OK ] MatMulIntegerToFloat.HasZeroPoint_NoBias_test_S8U8 (452 ms) [ RUN ] MatMulIntegerToFloat.NoZeroPoint_HasBias_test_S8U8 [ OK ] MatMulIntegerToFloat.NoZeroPoint_HasBias_test_S8U8 (454 ms) [ RUN ] MatMulIntegerToFloat.NoZeroPoint_NoBias_test_S8U8 [ OK ] MatMulIntegerToFloat.NoZeroPoint_NoBias_test_S8U8 (446 ms) [ RUN ] MatMulIntegerToFloat.HasZeroPoint_HasBias_test_S8U8 [ OK ] MatMulIntegerToFloat.HasZeroPoint_HasBias_test_S8U8 (508 ms) [ RUN ] MatMulIntegerToFloat.HasZeroPoint_NoBias_test_U8S8 [ OK ] MatMulIntegerToFloat.HasZeroPoint_NoBias_test_U8S8 (456 ms) [ RUN ] MatMulIntegerToFloat.NoZeroPoint_HasBias_test_U8S8 [ OK ] MatMulIntegerToFloat.NoZeroPoint_HasBias_test_U8S8 (455 ms) [ RUN ] MatMulIntegerToFloat.NoZeroPoint_NoBias_test_U8S8 [ OK ] MatMulIntegerToFloat.NoZeroPoint_NoBias_test_U8S8 (447 ms) [ RUN ] MatMulIntegerToFloat.HasZeroPoint_HasBias_test_U8S8 [ OK ] MatMulIntegerToFloat.HasZeroPoint_HasBias_test_U8S8 (465 ms) [ RUN ] MatMulIntegerToFloat.MatMulIntegerToFloat_FP16_U8U8 [ OK ] MatMulIntegerToFloat.MatMulIntegerToFloat_FP16_U8U8 (111 ms) [ RUN ] MatMulIntegerToFloat.MatMulIntegerToFloat_FP16_U8S8 [ OK ] MatMulIntegerToFloat.MatMulIntegerToFloat_FP16_U8S8 (115 ms) [ RUN ] MatMulIntegerToFloat.MatMulIntegerToFloat_FP16_S8S8 [ OK ] MatMulIntegerToFloat.MatMulIntegerToFloat_FP16_S8S8 (114 ms) [ RUN ] MatMulIntegerToFloat.MatMulIntegerToFloat_FP16_S8U8 [ OK ] MatMulIntegerToFloat.MatMulIntegerToFloat_FP16_S8U8 (110 ms) [ RUN ] MatMulIntegerToFloat.MatMulIntegerToFloat_FP16 [ OK ] MatMulIntegerToFloat.MatMulIntegerToFloat_FP16 (112 ms) [ RUN ] MatMulIntegerToFloat.MatMulInteger_With_ZeroPoint [ OK ] MatMulIntegerToFloat.MatMulInteger_With_ZeroPoint (337 ms) [----------] 22 tests from MatMulIntegerToFloat (8679 ms total) [----------] Global test environment tear-down [==========] 22 tests from 1 test suite ran. (8680 ms total) [ PASSED ] 22 tests. memleakdbg: ----- No memory leaks detected ----- ``` ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> * `CalculateMatMulIntegerToFloat` to replace CPU EP run reference * Added more FP32 testcases to isolate all input datatype combinations * Added fixed input to `MatMulIntegerToFloat_FP16*` test cases as for FP16 test cases. * onnxruntime/test/testdata/matmul_integer_to_float.py` is capable of generating FP16 models, but we do not produce any for now
This commit is contained in:
raoanag
GitHub
parent
2e13d5f0ab
commit
27b1dc91ab
23 changed files with 666 additions and 146 deletions
|
|
@ -2795,7 +2795,7 @@ This version of the operator has been available since version 1 of the 'com.micr
|
|||
<dd>Constrain input A data type to 8-bit integer tensor.</dd>
|
||||
<dt><tt>T2</tt> : tensor(int8), tensor(uint8)</dt>
|
||||
<dd>Constrain input B data type to 8-bit integer tensor.</dd>
|
||||
<dt><tt>T3</tt> : tensor(float)</dt>
|
||||
<dt><tt>T3</tt> : tensor(float), tensor(float16)</dt>
|
||||
<dd>Constrain input a_scale, b_scale and output Y data type as float tensor.</dd>
|
||||
</dl>
|
||||
|
||||
|
|
|
|||
|
|
@ -1268,6 +1268,7 @@ Do not modify directly.*
|
|||
|FusedMatMulActivation|*in* A:**T**<br> *in* B:**T**<br> *out* Y:**T**|1+|**T** = tensor(float), tensor(float16)|
|
||||
|Gelu|*in* X:**T**<br> *out* Y:**T**|1+|**T** = tensor(float), tensor(float16)|
|
||||
|GroupNorm|*in* X:**T**<br> *in* gamma:**M**<br> *in* beta:**M**<br> *out* Y:**T**|1+|**M** = tensor(float), tensor(float16)<br/> **T** = tensor(float), tensor(float16)|
|
||||
|MatMulIntegerToFloat|*in* A:**T1**<br> *in* B:**T2**<br> *in* a_scale:**T3**<br> *in* b_scale:**T3**<br> *in* a_zero_point:**T1**<br> *in* b_zero_point:**T2**<br> *in* bias:**T3**<br> *out* Y:**T3**|1+|**T1** = tensor(int8), tensor(uint8)<br/> **T2** = tensor(int8), tensor(uint8)<br/> **T3** = tensor(float), tensor(float16)|
|
||||
|MultiHeadAttention|*in* query:**T**<br> *in* key:**T**<br> *in* value:**T**<br> *in* bias:**T**<br> *in* key_padding_mask:**M**<br> *in* relative_position_bias:**T**<br> *in* past_key:**T**<br> *in* past_value:**T**<br> *out* output:**T**<br> *out* present_key:**T**<br> *out* present_value:**T**|1+|**M** = tensor(int32)<br/> **T** = tensor(float), tensor(float16)|
|
||||
|NhwcConv|*in* X:**T**<br> *in* W:**T**<br> *in* B:**T**<br> *out* Y:**T**|1+|**T** = tensor(float), tensor(float16)|
|
||||
|QLinearAdd|*in* A:**T**<br> *in* A_scale:**tensor(float)**<br> *in* A_zero_point:**T**<br> *in* B:**T**<br> *in* B_scale:**tensor(float)**<br> *in* B_zero_point:**T**<br> *in* C_scale:**tensor(float)**<br> *in* C_zero_point:**T**<br> *out* C:**T**|1+|**T** = tensor(int8), tensor(uint8)|
|
||||
|
|
|
|||
|
|
@ -434,7 +434,7 @@ ONNX_MS_OPERATOR_SET_SCHEMA(
|
|||
.Output(0, "Y", "Matrix multiply results from A * B", "T3")
|
||||
.TypeConstraint("T1", {"tensor(int8)", "tensor(uint8)"}, "Constrain input A data type to 8-bit integer tensor.")
|
||||
.TypeConstraint("T2", {"tensor(int8)", "tensor(uint8)"}, "Constrain input B data type to 8-bit integer tensor.")
|
||||
.TypeConstraint("T3", {"tensor(float)"},
|
||||
.TypeConstraint("T3", {"tensor(float)", "tensor(float16)"},
|
||||
"Constrain input a_scale, b_scale and output Y data type as float tensor.")
|
||||
.TypeAndShapeInferenceFunction([](ONNX_NAMESPACE::InferenceContext& ctx) {
|
||||
propagateElemTypeFromInputToOutput(ctx, 2, 0);
|
||||
|
|
|
|||
|
|
@ -278,7 +278,8 @@ InlinedVector<std::unique_ptr<GraphTransformer>> GenerateTransformers(
|
|||
onnxruntime::kAclExecutionProvider,
|
||||
onnxruntime::kArmNNExecutionProvider,
|
||||
onnxruntime::kJsExecutionProvider};
|
||||
|
||||
const InlinedHashSet<std::string_view> cpu_dml_eps = {onnxruntime::kCpuExecutionProvider,
|
||||
onnxruntime::kDmlExecutionProvider};
|
||||
#ifdef MLAS_TARGET_AMD64_IX86
|
||||
const bool avx2_precision_mode =
|
||||
session_options.config_options.GetConfigOrDefault(kOrtSessionOptionsAvx2PrecisionMode, "0") == "1" && MlasPlatformU8S8Overflow();
|
||||
|
|
@ -296,7 +297,7 @@ InlinedVector<std::unique_ptr<GraphTransformer>> GenerateTransformers(
|
|||
}
|
||||
|
||||
transformers.emplace_back(std::make_unique<GemmActivationFusion>(cpu_ep));
|
||||
transformers.emplace_back(std::make_unique<MatMulIntegerToFloatFusion>(cpu_ep));
|
||||
transformers.emplace_back(std::make_unique<MatMulIntegerToFloatFusion>(cpu_dml_eps));
|
||||
transformers.emplace_back(std::make_unique<DynamicQuantizeMatMulFusion>(cpu_ep));
|
||||
|
||||
transformers.emplace_back(std::make_unique<ConvActivationFusion>(cpu_cuda_rocm_acl_armnn_js_eps));
|
||||
|
|
|
|||
|
|
@ -31,6 +31,24 @@ static bool CheckBiasShape(const TensorShapeProto* bias_shape) {
|
|||
return bias_last_dim > 1;
|
||||
}
|
||||
|
||||
bool HasElementDataType(const NodeArg& node_arg, int32_t data_type) {
|
||||
if (!node_arg.Exists()) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const auto* type_proto = node_arg.TypeAsProto();
|
||||
if (!type_proto) {
|
||||
return false;
|
||||
}
|
||||
|
||||
int32_t actual_data_type;
|
||||
if (!utils::TryGetElementDataType(*type_proto, actual_data_type)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
return data_type == actual_data_type;
|
||||
}
|
||||
|
||||
/**
|
||||
MatMulIntegerToFloatFusion will fuse subgraph like below into MatMulIntegerToFloat:
|
||||
|
||||
|
|
@ -63,9 +81,10 @@ Status MatMulIntegerToFloatFusion::ApplyImpl(Graph& graph, bool& modified, int g
|
|||
auto& mul_node = *node_ptr;
|
||||
|
||||
ORT_RETURN_IF_ERROR(Recurse(mul_node, modified, graph_level, logger));
|
||||
|
||||
const bool is_dml_ep = node_ptr->GetExecutionProviderType() == kDmlExecutionProvider;
|
||||
if (!graph_utils::IsSupportedOptypeVersionAndDomain(mul_node, "Mul", {7, 13, 14}) ||
|
||||
!graph_utils::IsSupportedProvider(mul_node, GetCompatibleExecutionProviders())) {
|
||||
!graph_utils::IsSupportedProvider(mul_node, GetCompatibleExecutionProviders()) ||
|
||||
(!is_dml_ep && HasElementDataType(*mul_node.InputDefs()[0], ONNX_NAMESPACE::TensorProto_DataType_FLOAT16))) {
|
||||
continue;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -879,6 +879,12 @@ struct OperatorDescTraits<DML_QUANTIZED_LINEAR_MATRIX_MULTIPLY_OPERATOR_DESC>
|
|||
static constexpr DML_OPERATOR_TYPE Type = DML_OPERATOR_QUANTIZED_LINEAR_MATRIX_MULTIPLY;
|
||||
};
|
||||
|
||||
template <>
|
||||
struct OperatorDescTraits<DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_DESC>
|
||||
{
|
||||
static constexpr DML_OPERATOR_TYPE Type = DML_OPERATOR_MATRIX_MULTIPLY_INTEGER_TO_FLOAT;
|
||||
};
|
||||
|
||||
template <>
|
||||
struct OperatorDescTraits<DML_CONVOLUTION_INTEGER_OPERATOR_DESC>
|
||||
{
|
||||
|
|
@ -1041,12 +1047,6 @@ struct OperatorDescTraits<DML_QUANTIZED_LINEAR_AVERAGE_POOLING_OPERATOR_DESC>
|
|||
static constexpr DML_OPERATOR_TYPE Type = DML_OPERATOR_QUANTIZED_LINEAR_AVERAGE_POOLING;
|
||||
};
|
||||
|
||||
template <>
|
||||
struct OperatorDescTraits<DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_DESC>
|
||||
{
|
||||
static constexpr DML_OPERATOR_TYPE Type = DML_OPERATOR_MATRIX_MULTIPLY_INTEGER_TO_FLOAT;
|
||||
};
|
||||
|
||||
template <>
|
||||
struct OperatorDescTraits<DML_ACTIVATION_ELU_OPERATOR_DESC>
|
||||
{
|
||||
|
|
|
|||
|
|
@ -1885,6 +1885,25 @@ constexpr DML_OPERATOR_SCHEMA DML_QUANTIZED_LINEAR_MATRIX_MULTIPLY_OPERATOR_SCHE
|
|||
DML_QUANTIZED_LINEAR_MATRIX_MULTIPLY_OPERATOR_SCHEMA_FIELDS,
|
||||
};
|
||||
|
||||
constexpr DML_SCHEMA_FIELD DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA_FIELDS[8] {
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "ATensor", false },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "AScaleTensor", false },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "AZeroPointTensor", true },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "BTensor", false },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "BScaleTensor", false },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "BZeroPointTensor", true },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "BiasTensor", true },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_OUTPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "OutputTensor", false },
|
||||
};
|
||||
|
||||
constexpr DML_OPERATOR_SCHEMA DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA {
|
||||
"DML_OPERATOR_MATRIX_MULTIPLY_INTEGER_TO_FLOAT",
|
||||
static_cast<DML_OPERATOR_TYPE>(DML_OPERATOR_MATRIX_MULTIPLY_INTEGER_TO_FLOAT),
|
||||
DML_SCHEMA_OPERATOR_SUPPORT_FLAG_NONE,
|
||||
8,
|
||||
DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA_FIELDS,
|
||||
};
|
||||
|
||||
constexpr DML_SCHEMA_FIELD DML_CONVOLUTION_INTEGER_OPERATOR_SCHEMA_FIELDS[11] {
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "InputTensor", false },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "InputZeroPointTensor", true },
|
||||
|
|
@ -2395,24 +2414,6 @@ constexpr DML_OPERATOR_SCHEMA DML_QUANTIZED_LINEAR_AVERAGE_POOLING_OPERATOR_SCHE
|
|||
DML_QUANTIZED_LINEAR_AVERAGE_POOLING_OPERATOR_SCHEMA_FIELDS,
|
||||
};
|
||||
|
||||
constexpr DML_SCHEMA_FIELD DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA_FIELDS[8] {
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "ATensor", false },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "AScaleTensor", false },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "AZeroPointTensor", true },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "BTensor", false },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "BScaleTensor", false },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "BZeroPointTensor", true },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "BiasTensor", true },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_OUTPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "OutputTensor", false },
|
||||
};
|
||||
|
||||
constexpr DML_OPERATOR_SCHEMA DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA {
|
||||
"DML_OPERATOR_MATRIX_MULTIPLY_INTEGER_TO_FLOAT",
|
||||
DML_OPERATOR_MATRIX_MULTIPLY_INTEGER_TO_FLOAT,
|
||||
DML_SCHEMA_OPERATOR_SUPPORT_FLAG_NONE,
|
||||
8,
|
||||
DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA_FIELDS,
|
||||
};
|
||||
constexpr DML_SCHEMA_FIELD DML_ACTIVATION_ELU_OPERATOR_SCHEMA_FIELDS[3] {
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_INPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "InputTensor", false },
|
||||
DML_SCHEMA_FIELD { DML_SCHEMA_FIELD_KIND_OUTPUT_TENSOR, DML_SCHEMA_FIELD_TYPE_TENSOR_DESC, "OutputTensor", false },
|
||||
|
|
|
|||
|
|
@ -1139,6 +1139,19 @@ inline std::vector<OperatorField> GetFields(const DML_QUANTIZED_LINEAR_MATRIX_MU
|
|||
OperatorField(&DML_QUANTIZED_LINEAR_MATRIX_MULTIPLY_OPERATOR_SCHEMA.Fields[8], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.OutputTensor))),
|
||||
};
|
||||
}
|
||||
inline std::vector<OperatorField> GetFields(const DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_DESC& desc)
|
||||
{
|
||||
return {
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[0], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.ATensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[1], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.AScaleTensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[2], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.AZeroPointTensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[3], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.BTensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[4], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.BScaleTensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[5], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.BZeroPointTensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[6], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.BiasTensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[7], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.OutputTensor))),
|
||||
};
|
||||
}
|
||||
inline std::vector<OperatorField> GetFields(const DML_CONVOLUTION_INTEGER_OPERATOR_DESC& desc)
|
||||
{
|
||||
return {
|
||||
|
|
@ -1487,19 +1500,6 @@ inline std::vector<OperatorField> GetFields(const DML_QUANTIZED_LINEAR_AVERAGE_P
|
|||
OperatorField(&DML_QUANTIZED_LINEAR_AVERAGE_POOLING_OPERATOR_SCHEMA.Fields[12], ToOperatorFieldType(static_cast<UINT>(desc.IncludePadding))),
|
||||
};
|
||||
}
|
||||
inline std::vector<OperatorField> GetFields(const DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_DESC& desc)
|
||||
{
|
||||
return {
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[0], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.ATensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[1], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.AScaleTensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[2], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.AZeroPointTensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[3], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.BTensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[4], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.BScaleTensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[5], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.BZeroPointTensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[6], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.BiasTensor))),
|
||||
OperatorField(&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA.Fields[7], ToOperatorFieldType(static_cast<const DML_TENSOR_DESC*>(desc.OutputTensor))),
|
||||
};
|
||||
}
|
||||
inline std::vector<OperatorField> GetFields(const DML_ACTIVATION_ELU_OPERATOR_DESC& desc)
|
||||
{
|
||||
return {
|
||||
|
|
@ -1829,6 +1829,7 @@ inline const DML_OPERATOR_SCHEMA& GetSchema(DML_OPERATOR_TYPE operatorType)
|
|||
case DML_OPERATOR_RESAMPLE1: return DML_RESAMPLE1_OPERATOR_SCHEMA;
|
||||
case DML_OPERATOR_MATRIX_MULTIPLY_INTEGER: return DML_MATRIX_MULTIPLY_INTEGER_OPERATOR_SCHEMA;
|
||||
case DML_OPERATOR_QUANTIZED_LINEAR_MATRIX_MULTIPLY: return DML_QUANTIZED_LINEAR_MATRIX_MULTIPLY_OPERATOR_SCHEMA;
|
||||
case DML_OPERATOR_MATRIX_MULTIPLY_INTEGER_TO_FLOAT: return DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA;
|
||||
case DML_OPERATOR_CONVOLUTION_INTEGER: return DML_CONVOLUTION_INTEGER_OPERATOR_SCHEMA;
|
||||
case DML_OPERATOR_QUANTIZED_LINEAR_CONVOLUTION: return DML_QUANTIZED_LINEAR_CONVOLUTION_OPERATOR_SCHEMA;
|
||||
case DML_OPERATOR_ELEMENT_WISE_BIT_AND: return DML_ELEMENT_WISE_BIT_AND_OPERATOR_SCHEMA;
|
||||
|
|
@ -1856,7 +1857,6 @@ inline const DML_OPERATOR_SCHEMA& GetSchema(DML_OPERATOR_TYPE operatorType)
|
|||
case DML_OPERATOR_DIAGONAL_MATRIX1: return DML_DIAGONAL_MATRIX1_OPERATOR_SCHEMA;
|
||||
case DML_OPERATOR_MULTIHEAD_ATTENTION: return DML_MULTIHEAD_ATTENTION_OPERATOR_SCHEMA;
|
||||
case DML_OPERATOR_QUANTIZED_LINEAR_AVERAGE_POOLING: return DML_QUANTIZED_LINEAR_AVERAGE_POOLING_OPERATOR_SCHEMA;
|
||||
case DML_OPERATOR_MATRIX_MULTIPLY_INTEGER_TO_FLOAT: return DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA;
|
||||
case DML_OPERATOR_ACTIVATION_ELU: return DML_ACTIVATION_ELU_OPERATOR_SCHEMA;
|
||||
case DML_OPERATOR_ACTIVATION_CELU: return DML_ACTIVATION_CELU_OPERATOR_SCHEMA;
|
||||
case DML_OPERATOR_ACTIVATION_HARDMAX: return DML_ACTIVATION_HARDMAX_OPERATOR_SCHEMA;
|
||||
|
|
@ -2360,6 +2360,10 @@ inline AbstractOperatorDesc ConvertOperatorDesc(const DML_OPERATOR_DESC& opDesc)
|
|||
return AbstractOperatorDesc(
|
||||
&DML_QUANTIZED_LINEAR_MATRIX_MULTIPLY_OPERATOR_SCHEMA,
|
||||
GetFields(*static_cast<const DML_QUANTIZED_LINEAR_MATRIX_MULTIPLY_OPERATOR_DESC*>(opDesc.Desc)));
|
||||
case DML_OPERATOR_MATRIX_MULTIPLY_INTEGER_TO_FLOAT:
|
||||
return AbstractOperatorDesc(
|
||||
&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA,
|
||||
GetFields(*static_cast<const DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_DESC*>(opDesc.Desc)));
|
||||
case DML_OPERATOR_CONVOLUTION_INTEGER:
|
||||
return AbstractOperatorDesc(
|
||||
&DML_CONVOLUTION_INTEGER_OPERATOR_SCHEMA,
|
||||
|
|
@ -2468,10 +2472,6 @@ inline AbstractOperatorDesc ConvertOperatorDesc(const DML_OPERATOR_DESC& opDesc)
|
|||
return AbstractOperatorDesc(
|
||||
&DML_QUANTIZED_LINEAR_AVERAGE_POOLING_OPERATOR_SCHEMA,
|
||||
GetFields(*static_cast<const DML_QUANTIZED_LINEAR_AVERAGE_POOLING_OPERATOR_DESC*>(opDesc.Desc)));
|
||||
case DML_OPERATOR_MATRIX_MULTIPLY_INTEGER_TO_FLOAT:
|
||||
return AbstractOperatorDesc(
|
||||
&DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_SCHEMA,
|
||||
GetFields(*static_cast<const DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_DESC*>(opDesc.Desc)));
|
||||
case DML_OPERATOR_ACTIVATION_ELU:
|
||||
return AbstractOperatorDesc(
|
||||
&DML_ACTIVATION_ELU_OPERATOR_SCHEMA,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,111 @@
|
|||
// Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
// Licensed under the MIT License.
|
||||
|
||||
#include "precomp.h"
|
||||
|
||||
namespace Dml
|
||||
{
|
||||
|
||||
class DmlOperatorMatMulIntegerToFloat : public DmlOperator
|
||||
{
|
||||
enum OrtInputTensors : uint32_t
|
||||
{
|
||||
ortA,
|
||||
ortB,
|
||||
ortAScale,
|
||||
ortBScale,
|
||||
ortAZeroPoint,
|
||||
ortBZeroPoint,
|
||||
ortBias,
|
||||
ortInputCount
|
||||
};
|
||||
|
||||
enum DmlInputIndex : uint32_t
|
||||
{
|
||||
dmlA,
|
||||
dmlAScale,
|
||||
dmlAZeroPoint,
|
||||
dmlB,
|
||||
dmlBScale,
|
||||
dmlBZeroPoint,
|
||||
dmlBias,
|
||||
dmlInputCount,
|
||||
};
|
||||
|
||||
public:
|
||||
DmlOperatorMatMulIntegerToFloat(const MLOperatorKernelCreationContext& kernelInfo)
|
||||
: DmlOperator(kernelInfo)
|
||||
{
|
||||
std::vector<std::optional<uint32_t>> inputIndices = { OrtInputTensors::ortA, OrtInputTensors::ortAScale, OrtInputTensors::ortAZeroPoint, OrtInputTensors::ortB, OrtInputTensors::ortBScale, OrtInputTensors::ortBZeroPoint, OrtInputTensors::ortBias };
|
||||
DmlOperator::Initialize(kernelInfo, inputIndices);
|
||||
|
||||
std::vector<DimensionType> inputShape0 = kernelInfo.GetTensorShapeDescription().GetInputTensorShape(OrtInputTensors::ortA);
|
||||
std::vector<DimensionType> inputShape1 = kernelInfo.GetTensorShapeDescription().GetInputTensorShape(OrtInputTensors::ortB);
|
||||
std::vector<DimensionType> outputShape = kernelInfo.GetTensorShapeDescription().GetOutputTensorShape(0);
|
||||
|
||||
OperatorHelper::MatMulShapeMapping(inputShape0, inputShape1, outputShape);
|
||||
|
||||
// Initialize the input descriptions with broadcasting
|
||||
m_inputTensorDescs[DmlInputIndex::dmlA] = CreateTensorDescFromInput(kernelInfo, OrtInputTensors::ortA, TensorAxis::DoNotCoerce, TensorAxis::W, TensorAxis::RightAligned, inputShape0);
|
||||
m_inputTensorDescs[DmlInputIndex::dmlB] = CreateTensorDescFromInput(kernelInfo, OrtInputTensors::ortB, TensorAxis::DoNotCoerce, TensorAxis::W, TensorAxis::RightAligned, inputShape1);
|
||||
|
||||
// Broadcast Bias tensor to the shape of the output tensor.
|
||||
if(kernelInfo.IsInputValid(OrtInputTensors::ortBias)) {
|
||||
m_inputTensorDescs[DmlInputIndex::dmlBias] = CreateTensorDescFromInput(kernelInfo, OrtInputTensors::ortBias, TensorAxis::DoNotCoerce,
|
||||
TensorAxis::W, TensorAxis::RightAligned, outputShape);
|
||||
}
|
||||
|
||||
uint32_t dmlDimSize = m_inputTensorDescs[DmlInputIndex::dmlA].GetDimensionCount();
|
||||
// Resize the A Scale to be the same dimension as the input tensor.
|
||||
// The 1D tensor needs to be moved to the H channel.
|
||||
m_inputTensorDescs[DmlInputIndex::dmlAScale] = CreateTensorDescFromInput(
|
||||
kernelInfo,
|
||||
OrtInputTensors::ortAScale,
|
||||
TensorAxis::DoNotCoerce,
|
||||
TensorAxis::H,
|
||||
TensorAxis::LeftAligned,
|
||||
std::nullopt,
|
||||
dmlDimSize
|
||||
);
|
||||
|
||||
// Resize the A ZeroPoint to be the same dimension as the input tensor.
|
||||
// The 1D tensor needs to be moved to the H channel.
|
||||
if (kernelInfo.IsInputValid(OrtInputTensors::ortAZeroPoint))
|
||||
{
|
||||
m_inputTensorDescs[DmlInputIndex::dmlAZeroPoint] = CreateTensorDescFromInput(
|
||||
kernelInfo,
|
||||
OrtInputTensors::ortAZeroPoint,
|
||||
TensorAxis::DoNotCoerce,
|
||||
TensorAxis::H,
|
||||
TensorAxis::LeftAligned,
|
||||
std::nullopt,
|
||||
dmlDimSize
|
||||
);
|
||||
}
|
||||
|
||||
// B Zeropoint and BScale are already aligned in the W dimension so no need to align them
|
||||
|
||||
// Initialize the output description while overriding the shape
|
||||
m_outputTensorDescs[0] = CreateTensorDescFromOutput(kernelInfo, 0, TensorAxis::DoNotCoerce, TensorAxis::W, TensorAxis::RightAligned, outputShape);
|
||||
|
||||
std::vector<DML_TENSOR_DESC> inputDescs = GetDmlInputDescs();
|
||||
std::vector<DML_TENSOR_DESC> outputDescs = GetDmlOutputDescs();
|
||||
|
||||
DML_MATRIX_MULTIPLY_INTEGER_TO_FLOAT_OPERATOR_DESC matMulDesc = {};
|
||||
matMulDesc.ATensor = &inputDescs[DmlInputIndex::dmlA];
|
||||
matMulDesc.AScaleTensor = &inputDescs[DmlInputIndex::dmlAScale];
|
||||
matMulDesc.AZeroPointTensor = inputDescs[DmlInputIndex::dmlAZeroPoint].Desc != nullptr ? &inputDescs[DmlInputIndex::dmlAZeroPoint] : nullptr;
|
||||
matMulDesc.BTensor = &inputDescs[DmlInputIndex::dmlB];
|
||||
matMulDesc.BScaleTensor = &inputDescs[DmlInputIndex::dmlBScale];
|
||||
matMulDesc.BZeroPointTensor = inputDescs[DmlInputIndex::dmlBZeroPoint].Desc != nullptr ? &inputDescs[DmlInputIndex::dmlBZeroPoint] : nullptr;
|
||||
matMulDesc.BiasTensor = inputDescs[DmlInputIndex::dmlBias].Desc != nullptr ? &inputDescs[DmlInputIndex::dmlBias] : nullptr;
|
||||
matMulDesc.OutputTensor = &outputDescs[0];
|
||||
|
||||
DML_OPERATOR_DESC opDesc = { (DML_OPERATOR_TYPE) DML_OPERATOR_MATRIX_MULTIPLY_INTEGER_TO_FLOAT, &matMulDesc };
|
||||
SetDmlOperatorDesc(opDesc, kernelInfo);
|
||||
}
|
||||
};
|
||||
|
||||
DML_OP_DEFINE_CREATION_FUNCTION(MatMulIntegerToFloat, DmlOperatorMatMulIntegerToFloat);
|
||||
|
||||
} // namespace Dml
|
||||
|
|
@ -503,6 +503,7 @@ DML_OP_EXTERN_CREATION_FUNCTION(QLinearMatMul);
|
|||
DML_OP_EXTERN_CREATION_FUNCTION(QLinearConcat);
|
||||
DML_OP_EXTERN_CREATION_FUNCTION(DynamicQuantizeLinear);
|
||||
DML_OP_EXTERN_CREATION_FUNCTION(MatMulInteger);
|
||||
DML_OP_EXTERN_CREATION_FUNCTION(MatMulIntegerToFloat);
|
||||
DML_OP_EXTERN_CREATION_FUNCTION(ConvInteger);
|
||||
DML_OP_EXTERN_CREATION_FUNCTION(Trilu);
|
||||
|
||||
|
|
@ -622,6 +623,13 @@ constexpr static std::array<SupportedTensorDataTypes, 3> supportedTypeListQLinea
|
|||
SupportedTensorDataTypes::Int8|SupportedTensorDataTypes::UInt8,
|
||||
SupportedTensorDataTypes::Int8|SupportedTensorDataTypes::UInt8
|
||||
};
|
||||
|
||||
constexpr static std::array<SupportedTensorDataTypes, 3> supportedTypeListMatMulIntegerToFloat = {
|
||||
SupportedTensorDataTypes::Ints8Bit,
|
||||
SupportedTensorDataTypes::Ints8Bit,
|
||||
SupportedTensorDataTypes::Float16to32
|
||||
};
|
||||
|
||||
constexpr static std::array<SupportedTensorDataTypes, 4> supportedTypeListQLinearConv = {
|
||||
SupportedTensorDataTypes::Int8|SupportedTensorDataTypes::UInt8,
|
||||
SupportedTensorDataTypes::Int8|SupportedTensorDataTypes::UInt8,
|
||||
|
|
@ -1083,6 +1091,7 @@ constexpr static OperatorRegistrationInformation operatorRegistrationInformation
|
|||
{REG_INFO( 10, QLinearConv, typeNameListFour, supportedTypeListQLinearConv, DmlGraphSupport::Supported)},
|
||||
{REG_INFO( 10, QLinearMatMul, typeNameListThree, supportedTypeListQLinearMatMul, DmlGraphSupport::Supported)},
|
||||
{REG_INFO( 10, MatMulInteger, typeNameListThree, supportedTypeListInteger, DmlGraphSupport::Supported)},
|
||||
{REG_INFO_MS( 1, MatMulIntegerToFloat, typeNameListThree, supportedTypeListMatMulIntegerToFloat, DmlGraphSupport::Supported)},
|
||||
{REG_INFO( 10, ConvInteger, typeNameListThree, supportedTypeListInteger, DmlGraphSupport::Supported)},
|
||||
{REG_INFO( 11, DynamicQuantizeLinear, typeNameListTwo, supportedTypeListDynamicQuantizeLinear, DmlGraphSupport::Supported)},
|
||||
{REG_INFO( 7, LayerNormalization, typeNameListLayerNormContrib, supportedTypeListLayerNormalizationContrib, DmlGraphSupport::Supported, requiredConstantCpuInputs(), std::nullopt, QueryLayerNormalization)},
|
||||
|
|
|
|||
|
|
@ -870,7 +870,6 @@ public:
|
|||
QLinearMatMulHelper(const Info_t& info, const Shape_t& shape) : MatMulHelperBase(info, shape, 0, 3) {}
|
||||
};
|
||||
|
||||
|
||||
class TopKHelper
|
||||
{
|
||||
void Initialize(
|
||||
|
|
@ -1776,6 +1775,7 @@ using ShapeInferenceHelper_Identity16 = GetOutputShapeAsInputShapeHelper;
|
|||
using ShapeInferenceHelper_Identity19 = GetOutputShapeAsInputShapeHelper;
|
||||
using ShapeInferenceHelper_MatMul = MatMulHelper;
|
||||
using ShapeInferenceHelper_MatMulInteger = MatMulHelper;
|
||||
using ShapeInferenceHelper_MatMulIntegerToFloat = MatMulHelper;
|
||||
using ShapeInferenceHelper_QLinearMatMul = QLinearMatMulHelper;
|
||||
using ShapeInferenceHelper_QLinearAdd = GetBroadcastedOutputShapeHelper;
|
||||
using ShapeInferenceHelper_DynamicQuantizeLinear = GetOutputShapeAsInputShapeHelper;
|
||||
|
|
|
|||
|
|
@ -449,6 +449,7 @@ namespace OperatorHelper
|
|||
static const int sc_sinceVer_FusedMatMulActivation = 1;
|
||||
static const int sc_sinceVer_QLinearSigmoid = 1;
|
||||
static const int sc_sinceVer_Attention = 1;
|
||||
static const int sc_sinceVer_MatMulIntegerToFloat = 1;
|
||||
static const int sc_sinceVer_MultiHeadAttention = 1;
|
||||
static const int sc_sinceVer_SkipLayerNormalization = 1;
|
||||
static const int sc_sinceVer_EmbedLayerNormalization = 1;
|
||||
|
|
|
|||
|
|
@ -23,136 +23,408 @@ using namespace std;
|
|||
namespace onnxruntime {
|
||||
namespace test {
|
||||
|
||||
template <typename IType, typename WType>
|
||||
void TestMatMulIntegerToFloat(const std::vector<int64_t>& A_dims,
|
||||
std::vector<int64_t> B_dims,
|
||||
const std::string& reference_model,
|
||||
bool is_matrix_b_constant,
|
||||
template <typename IType, typename WType, typename OType>
|
||||
static void CalculateMatMulIntegerToFloat(const int64_t M, const int64_t N, const int64_t K,
|
||||
const std::vector<IType>& A_data, const std::vector<OType>& A_scale,
|
||||
const std::vector<IType>& A_zero_point, const std::vector<WType>& B_data,
|
||||
std::vector<OType>& B_scale, std::vector<WType>& B_zero_point,
|
||||
const std::vector<OType>& Bias, std::vector<float>& Y_data,
|
||||
bool per_column, bool has_zp, bool has_bias) {
|
||||
if (!per_column) {
|
||||
B_zero_point.resize(N, B_zero_point[0]);
|
||||
B_scale.resize(N, B_scale[0]);
|
||||
}
|
||||
|
||||
for (int64_t m = 0; m < M; m++) {
|
||||
for (int64_t n = 0; n < N; n++) {
|
||||
float sum = 0.0f;
|
||||
for (int64_t k = 0; k < K; k++) {
|
||||
float A_dequantized = has_zp ? (static_cast<int>(A_data[m * K + k]) - static_cast<int>(A_zero_point[0])) * A_scale[0] : A_data[m * K + k] * A_scale[0];
|
||||
float B_dequantized = has_zp ? (static_cast<int>(B_data[k * N + n]) - static_cast<int>(B_zero_point[n])) * B_scale[n] : B_data[k * N + n] * B_scale[n];
|
||||
|
||||
sum += A_dequantized * B_dequantized;
|
||||
}
|
||||
if (has_bias) {
|
||||
sum += Bias[n];
|
||||
}
|
||||
Y_data[m * N + n] = static_cast<OType>(sum);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
template <typename IType, typename WType, typename OType>
|
||||
void TestMatMulIntegerToFloat(bool is_matrix_b_constant,
|
||||
bool per_column = false,
|
||||
bool has_zp = true,
|
||||
bool has_bias = false) {
|
||||
// create rand inputs
|
||||
RandomValueGenerator random{};
|
||||
|
||||
int64_t M = 4;
|
||||
int64_t N = 128;
|
||||
int64_t K = 128;
|
||||
std::vector<int64_t> A_dims{M, K};
|
||||
std::vector<int64_t> B_dims{K, N};
|
||||
std::vector<int64_t> Y_dims{M, K};
|
||||
std::vector<IType> A_data;
|
||||
std::vector<int> tmp_A_data = random.Uniform<int32_t>(A_dims,
|
||||
std::numeric_limits<WType>::lowest(),
|
||||
std::numeric_limits<WType>::max());
|
||||
std::transform(tmp_A_data.begin(), tmp_A_data.end(), std::back_inserter(A_data), [](int32_t v) -> WType {
|
||||
std::vector<IType> tmp_A_data = random.Uniform<IType>(A_dims,
|
||||
std::numeric_limits<IType>::lowest(),
|
||||
std::numeric_limits<IType>::max());
|
||||
std::transform(tmp_A_data.begin(), tmp_A_data.end(), std::back_inserter(A_data), [](int32_t v) -> IType {
|
||||
return static_cast<IType>(v);
|
||||
});
|
||||
|
||||
std::vector<WType> B_data;
|
||||
std::vector<int> tmp_B_data = random.Uniform<int32_t>(B_dims,
|
||||
std::numeric_limits<WType>::lowest(),
|
||||
std::numeric_limits<WType>::max());
|
||||
|
||||
std::vector<WType> tmp_B_data;
|
||||
tmp_B_data = random.Uniform<WType>(B_dims,
|
||||
std::is_signed<WType>::value ? std::numeric_limits<int8_t>::lowest() / 2 : std::numeric_limits<uint8_t>::lowest(),
|
||||
std::numeric_limits<WType>::max() / 2);
|
||||
std::transform(tmp_B_data.begin(), tmp_B_data.end(), std::back_inserter(B_data), [](int32_t v) -> WType {
|
||||
return static_cast<WType>(v);
|
||||
});
|
||||
|
||||
std::vector<float> A_scale = random.Uniform<float>(AsSpan<int64_t>({1}), -0.1f, 0.1f);
|
||||
std::vector<OType> A_scale = random.Uniform<OType>(AsSpan<int64_t>({1}), -0.1f, 0.1f);
|
||||
std::vector<IType> A_zero_point{(std::numeric_limits<IType>::lowest() + std::numeric_limits<IType>::max() + IType(2)) / 2};
|
||||
|
||||
int64_t b_scale_zp_size = per_column ? B_dims.back() : 1;
|
||||
std::vector<float> B_scale = random.Uniform<float>(AsSpan({b_scale_zp_size}), -0.1f, 0.1f);
|
||||
std::vector<OType> B_scale = random.Uniform<OType>(AsSpan({b_scale_zp_size}), -0.1f, 0.1f);
|
||||
|
||||
std::vector<WType> B_zero_point(b_scale_zp_size);
|
||||
std::for_each(B_zero_point.begin(),
|
||||
B_zero_point.end(),
|
||||
[&random](WType& zp) {
|
||||
zp = static_cast<WType>(random.Uniform<int32_t>(std::array<int64_t, 1>{1},
|
||||
std::numeric_limits<WType>::lowest(),
|
||||
std::numeric_limits<WType>::max())[0]);
|
||||
zp = static_cast<WType>(random.Uniform<WType>(std::array<int64_t, 1>{1},
|
||||
std::numeric_limits<WType>::lowest(),
|
||||
std::numeric_limits<WType>::max())[0]);
|
||||
});
|
||||
|
||||
std::vector<float> Bias = random.Uniform<float>(AsSpan({B_dims.back()}), -0.1f, 0.1f);
|
||||
std::vector<OType> Bias = random.Uniform<OType>(AsSpan({B_dims.back()}), -0.1f, 0.1f);
|
||||
|
||||
OpTester test("MatMulIntegerToFloat", 1, onnxruntime::kMSDomain);
|
||||
test.AddInput<IType>("A", A_dims, A_data);
|
||||
test.AddInput<WType>("B", B_dims, B_data, is_matrix_b_constant);
|
||||
test.AddInput<float>("a_scale", {1}, A_scale);
|
||||
test.AddInput<float>("b_scale", {b_scale_zp_size}, B_scale);
|
||||
test.AddInput<OType>("a_scale", {1}, A_scale);
|
||||
test.AddInput<OType>("b_scale", {b_scale_zp_size}, B_scale);
|
||||
|
||||
if (has_zp) {
|
||||
test.AddInput<IType>("a_zero_point", {1}, A_zero_point);
|
||||
test.AddInput<WType>("b_zero_point", {b_scale_zp_size}, B_zero_point);
|
||||
} else {
|
||||
test.AddOptionalInputEdge<WType>();
|
||||
test.AddOptionalInputEdge<IType>();
|
||||
test.AddOptionalInputEdge<WType>();
|
||||
}
|
||||
|
||||
if (has_bias) {
|
||||
test.AddInput<float>("bias", {B_dims.back()}, Bias);
|
||||
test.AddInput<OType>("bias", {B_dims.back()}, Bias);
|
||||
} else {
|
||||
test.AddOptionalInputEdge<float>();
|
||||
test.AddOptionalInputEdge<OType>();
|
||||
}
|
||||
|
||||
test.AddReferenceOutputs(reference_model);
|
||||
test.SetOutputRelErr("Y", 1e-4f);
|
||||
test.Run();
|
||||
std::vector<float> Y_data(M * N);
|
||||
CalculateMatMulIntegerToFloat<IType, WType, OType>(M, N, K, A_data, A_scale, A_zero_point,
|
||||
B_data, B_scale, B_zero_point, Bias, Y_data,
|
||||
per_column, has_zp, has_bias);
|
||||
|
||||
if (std::is_same_v<OType, float>) {
|
||||
test.AddOutput<float>("Y", {M, N}, Y_data);
|
||||
test.SetOutputRelErr("Y", 0.02f);
|
||||
} else {
|
||||
test.AddOutput<MLFloat16>("Y", {M, N}, ToFloat16(Y_data));
|
||||
test.SetOutputAbsErr("Y", 0.5f);
|
||||
}
|
||||
|
||||
// Only DML EP supports these data type combinations for now
|
||||
if (std::is_same_v<OType, MLFloat16> ||
|
||||
(std::is_same_v<OType, float> &&
|
||||
std::is_same_v<IType, int8_t> &&
|
||||
std::is_same_v<WType, uint8_t>)) {
|
||||
std::vector<std::unique_ptr<IExecutionProvider>> execution_providers;
|
||||
execution_providers.push_back(DefaultDmlExecutionProvider());
|
||||
test.Run(OpTester::ExpectResult::kExpectSuccess, "", {}, nullptr, &execution_providers);
|
||||
} else {
|
||||
test.Run();
|
||||
}
|
||||
}
|
||||
|
||||
template <typename IType, typename WType, bool HasZeroPoint, bool HasBias>
|
||||
void RunMatMulIntegerToFloatTest(const string& model_path) {
|
||||
std::vector<int64_t> A_dims{4, 128};
|
||||
std::vector<int64_t> B_dims{128, 128};
|
||||
std::vector<int64_t> Y_dims{4, 128};
|
||||
|
||||
TestMatMulIntegerToFloat<IType, WType>(A_dims,
|
||||
B_dims,
|
||||
model_path,
|
||||
false, /*is_matrix_b_constant*/
|
||||
false, /*per_column*/
|
||||
HasZeroPoint, /*has_zp*/
|
||||
HasBias /*has_bias*/
|
||||
template <typename IType, typename WType, typename OType, bool HasZeroPoint, bool HasBias>
|
||||
void RunMatMulIntegerToFloatTest() {
|
||||
TestMatMulIntegerToFloat<IType, WType, OType>(
|
||||
false, /*is_matrix_b_constant*/
|
||||
false, /*per_column*/
|
||||
HasZeroPoint, /*has_zp*/
|
||||
HasBias /*has_bias*/
|
||||
);
|
||||
|
||||
TestMatMulIntegerToFloat<IType, WType>(A_dims,
|
||||
B_dims,
|
||||
model_path,
|
||||
true, /*is_matrix_b_constant*/
|
||||
false, /*per_column*/
|
||||
HasZeroPoint, /*has_zp*/
|
||||
HasBias /*has_bias*/
|
||||
TestMatMulIntegerToFloat<IType, WType, OType>(
|
||||
true, /*is_matrix_b_constant*/
|
||||
false, /*per_column*/
|
||||
HasZeroPoint, /*has_zp*/
|
||||
HasBias /*has_bias*/
|
||||
);
|
||||
|
||||
TestMatMulIntegerToFloat<IType, WType>(A_dims,
|
||||
B_dims,
|
||||
model_path,
|
||||
false, /*is_matrix_b_constant*/
|
||||
true, /*per_column*/
|
||||
HasZeroPoint, /*has_zp*/
|
||||
HasBias /*has_bias*/
|
||||
TestMatMulIntegerToFloat<IType, WType, OType>(
|
||||
false, /*is_matrix_b_constant*/
|
||||
true, /*per_column*/
|
||||
HasZeroPoint, /*has_zp*/
|
||||
HasBias /*has_bias*/
|
||||
);
|
||||
|
||||
TestMatMulIntegerToFloat<IType, WType>(A_dims,
|
||||
B_dims,
|
||||
model_path,
|
||||
true, /*is_matrix_b_constant*/
|
||||
true, /*per_column*/
|
||||
HasZeroPoint, /*has_zp*/
|
||||
HasBias /*has_bias*/
|
||||
TestMatMulIntegerToFloat<IType, WType, OType>(
|
||||
true, /*is_matrix_b_constant*/
|
||||
true, /*per_column*/
|
||||
HasZeroPoint, /*has_zp*/
|
||||
HasBias /*has_bias*/
|
||||
);
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, HasZeroPoint_NoBias_test_U8X8) {
|
||||
RunMatMulIntegerToFloatTest<uint8_t, int8_t, true, false>("testdata/matmul_integer_to_float_int8.onnx");
|
||||
RunMatMulIntegerToFloatTest<uint8_t, uint8_t, true, false>("testdata/matmul_integer_to_float_uint8.onnx");
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, NoZeroPoint_HasBias_test_U8X8) {
|
||||
RunMatMulIntegerToFloatTest<uint8_t, int8_t, false, true>("testdata/matmul_integer_to_float_int8_bias.onnx");
|
||||
RunMatMulIntegerToFloatTest<uint8_t, uint8_t, false, true>("testdata/matmul_integer_to_float_uint8_bias.onnx");
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, HasZeroPoint_NoBias_test_S8S8) {
|
||||
RunMatMulIntegerToFloatTest<int8_t, int8_t, true, false>("testdata/matmul_integer_to_float_int8_int8.onnx");
|
||||
RunMatMulIntegerToFloatTest<int8_t, int8_t, float, true, false>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, NoZeroPoint_HasBias_test_S8S8) {
|
||||
RunMatMulIntegerToFloatTest<int8_t, int8_t, false, true>("testdata/matmul_integer_to_float_int8_int8_bias.onnx");
|
||||
RunMatMulIntegerToFloatTest<int8_t, int8_t, float, false, true>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, NoZeroPoint_NoBias_test_S8S8) {
|
||||
RunMatMulIntegerToFloatTest<int8_t, int8_t, float, false, false>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, HasZeroPoint_HasBias_test_S8S8) {
|
||||
RunMatMulIntegerToFloatTest<int8_t, int8_t, float, true, true>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, HasZeroPoint_NoBias_test_U8U8) {
|
||||
RunMatMulIntegerToFloatTest<uint8_t, uint8_t, float, true, false>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, NoZeroPoint_HasBias_test_U8U8) {
|
||||
RunMatMulIntegerToFloatTest<uint8_t, uint8_t, float, false, true>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, NoZeroPoint_NoBias_test_U8U8) {
|
||||
RunMatMulIntegerToFloatTest<uint8_t, uint8_t, float, false, false>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, HasZeroPoint_HasBias_test_U8X8) {
|
||||
RunMatMulIntegerToFloatTest<uint8_t, uint8_t, float, true, true>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, HasZeroPoint_NoBias_test_U8S8) {
|
||||
RunMatMulIntegerToFloatTest<uint8_t, int8_t, float, true, false>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, NoZeroPoint_HasBias_test_U8S8) {
|
||||
RunMatMulIntegerToFloatTest<uint8_t, int8_t, float, false, true>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, NoZeroPoint_NoBias_test_U8S8) {
|
||||
RunMatMulIntegerToFloatTest<uint8_t, int8_t, float, false, false>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, HasZeroPoint_HasBias_test_U8S8) {
|
||||
RunMatMulIntegerToFloatTest<uint8_t, int8_t, float, true, true>();
|
||||
}
|
||||
|
||||
// DML EP supports Float16 output type and Signed A Matrix and Unsigned B Matric for Float32 output
|
||||
#if defined(USE_DML)
|
||||
|
||||
TEST(MatMulIntegerToFloat, HasZeroPoint_NoBias_test_S8U8) {
|
||||
RunMatMulIntegerToFloatTest<int8_t, uint8_t, float, true, false>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, NoZeroPoint_HasBias_test_S8U8) {
|
||||
RunMatMulIntegerToFloatTest<int8_t, uint8_t, float, false, true>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, NoZeroPoint_NoBias_test_S8U8) {
|
||||
RunMatMulIntegerToFloatTest<int8_t, uint8_t, float, false, false>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, HasZeroPoint_HasBias_test_S8U8) {
|
||||
RunMatMulIntegerToFloatTest<int8_t, int8_t, float, true, true>();
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, MatMulIntegerToFloat_FP16_U8U8) {
|
||||
OpTester test("MatMulIntegerToFloat", 1, kMSDomain);
|
||||
int64_t M = 5;
|
||||
int64_t N = 5;
|
||||
int64_t K = 2;
|
||||
|
||||
std::vector<uint8_t> A_data = {1, 5, 2, 1, 9,
|
||||
1, 1, 3, 7, 2};
|
||||
std::vector<uint8_t> B_data = {3, 7, 2, 1, 1,
|
||||
2, 1, 9, 1, 1};
|
||||
std::vector<MLFloat16> A_scale = ToFloat16({3.0f});
|
||||
std::vector<MLFloat16> B_scale = ToFloat16({2.0f});
|
||||
test.AddInput<uint8_t>("A", {M, K}, A_data);
|
||||
test.AddInput<uint8_t>("B", {K, N}, B_data);
|
||||
std::vector<uint8_t> A_zero_point = {1};
|
||||
std::vector<uint8_t> B_zero_point = {1};
|
||||
|
||||
test.AddInput<MLFloat16>("a_scale", {1}, A_scale);
|
||||
test.AddInput<MLFloat16>("b_scale", {1}, B_scale);
|
||||
test.AddInput<uint8_t>("a_zero_point", {1}, A_zero_point);
|
||||
test.AddInput<uint8_t>("b_zero_point", {1}, B_zero_point);
|
||||
|
||||
std::vector<float> Y_data(M * N);
|
||||
CalculateMatMulIntegerToFloat<uint8_t, uint8_t, MLFloat16>(M, N, K, A_data, A_scale, A_zero_point,
|
||||
B_data, B_scale, B_zero_point, {}, Y_data,
|
||||
false, true, false);
|
||||
|
||||
test.AddOutput<MLFloat16>("Y", {M, N}, ToFloat16(Y_data));
|
||||
std::vector<std::unique_ptr<IExecutionProvider>> execution_providers;
|
||||
execution_providers.push_back(DefaultDmlExecutionProvider());
|
||||
test.Run(OpTester::ExpectResult::kExpectSuccess, "", {}, nullptr, &execution_providers);
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, MatMulIntegerToFloat_FP16_U8S8) {
|
||||
OpTester test("MatMulIntegerToFloat", 1, kMSDomain);
|
||||
int64_t M = 5;
|
||||
int64_t N = 5;
|
||||
int64_t K = 2;
|
||||
|
||||
std::vector<uint8_t> A_data = {3, 7, 2, 1, 1,
|
||||
2, 1, 9, 1, 1};
|
||||
std::vector<int8_t> B_data = {2, -1, -9, 1, 1,
|
||||
-1, 0, -3, 1, -4};
|
||||
std::vector<MLFloat16> A_scale = ToFloat16({-4.0f});
|
||||
std::vector<MLFloat16> B_scale = ToFloat16({2.0f});
|
||||
test.AddInput<uint8_t>("A", {M, K}, A_data);
|
||||
test.AddInput<int8_t>("B", {K, N}, B_data);
|
||||
std::vector<uint8_t> A_zero_point = {1};
|
||||
std::vector<int8_t> B_zero_point = {3};
|
||||
std::vector<MLFloat16> Bias = ToFloat16({11.0f, -17.0f, 1.0f, -3.0f, 12.0f});
|
||||
|
||||
test.AddInput<MLFloat16>("a_scale", {1}, A_scale);
|
||||
test.AddInput<MLFloat16>("b_scale", {1}, B_scale);
|
||||
test.AddInput<uint8_t>("a_zero_point", {1}, A_zero_point);
|
||||
test.AddInput<int8_t>("b_zero_point", {1}, B_zero_point);
|
||||
|
||||
std::vector<float> Y_data(M * N);
|
||||
CalculateMatMulIntegerToFloat<uint8_t, int8_t, MLFloat16>(M, N, K, A_data, A_scale, A_zero_point,
|
||||
B_data, B_scale, B_zero_point, {}, Y_data,
|
||||
false, true, false);
|
||||
|
||||
test.AddOutput<MLFloat16>("Y", {M, N}, ToFloat16(Y_data));
|
||||
|
||||
std::vector<std::unique_ptr<IExecutionProvider>> execution_providers;
|
||||
execution_providers.push_back(DefaultDmlExecutionProvider());
|
||||
test.Run(OpTester::ExpectResult::kExpectSuccess, "", {}, nullptr, &execution_providers);
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, MatMulIntegerToFloat_FP16_S8S8) {
|
||||
OpTester test("MatMulIntegerToFloat", 1, kMSDomain);
|
||||
int64_t M = 5;
|
||||
int64_t N = 5;
|
||||
int64_t K = 2;
|
||||
|
||||
std::vector<int8_t> A_data = {3, 7, -2, 1, 1,
|
||||
2, -1, -9, 1, 1};
|
||||
std::vector<int8_t> B_data = {2, -1, -9, 1, 1,
|
||||
-1, 0, -3, 1, -4};
|
||||
std::vector<MLFloat16> A_scale = ToFloat16({-4.0f});
|
||||
std::vector<MLFloat16> B_scale = ToFloat16({2.0f});
|
||||
test.AddInput<int8_t>("A", {M, K}, A_data);
|
||||
test.AddInput<int8_t>("B", {K, N}, B_data);
|
||||
std::vector<int8_t> A_zero_point = {-1};
|
||||
std::vector<int8_t> B_zero_point = {3};
|
||||
std::vector<MLFloat16> Bias = ToFloat16({11.0f, -17.0f, 1.0f, -3.0f, 12.0f});
|
||||
|
||||
test.AddInput<MLFloat16>("a_scale", {1}, A_scale);
|
||||
test.AddInput<MLFloat16>("b_scale", {1}, B_scale);
|
||||
test.AddInput<int8_t>("a_zero_point", {1}, A_zero_point);
|
||||
test.AddInput<int8_t>("b_zero_point", {1}, B_zero_point);
|
||||
test.AddInput<MLFloat16>("bias", {N}, Bias);
|
||||
|
||||
std::vector<float> Y_data(M * N);
|
||||
CalculateMatMulIntegerToFloat<int8_t, int8_t, MLFloat16>(M, N, K, A_data, A_scale, A_zero_point,
|
||||
B_data, B_scale, B_zero_point, Bias, Y_data,
|
||||
false, true, true);
|
||||
|
||||
test.AddOutput<MLFloat16>("Y", {M, N}, ToFloat16(Y_data));
|
||||
|
||||
std::vector<std::unique_ptr<IExecutionProvider>> execution_providers;
|
||||
execution_providers.push_back(DefaultDmlExecutionProvider());
|
||||
test.Run(OpTester::ExpectResult::kExpectSuccess, "", {}, nullptr, &execution_providers);
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, MatMulIntegerToFloat_FP16_S8U8) {
|
||||
OpTester test("MatMulIntegerToFloat", 1, kMSDomain);
|
||||
int64_t M = 5;
|
||||
int64_t N = 5;
|
||||
int64_t K = 2;
|
||||
|
||||
std::vector<int8_t> A_data = {3, 7, -2, 1, 1,
|
||||
2, -1, -9, 1, 1};
|
||||
std::vector<uint8_t> B_data = {3, 7, 2, 1, 1,
|
||||
2, 1, 9, 1, 1};
|
||||
std::vector<MLFloat16> A_scale = ToFloat16({-4.0f});
|
||||
std::vector<MLFloat16> B_scale = ToFloat16({2.0f});
|
||||
test.AddInput<int8_t>("A", {M, K}, A_data);
|
||||
test.AddInput<uint8_t>("B", {K, N}, B_data);
|
||||
std::vector<int8_t> A_zero_point = {-1};
|
||||
std::vector<uint8_t> B_zero_point = {1};
|
||||
std::vector<MLFloat16> Bias = ToFloat16({11.0f, -17.0f, 1.0f, -3.0f, 12.0f});
|
||||
|
||||
test.AddInput<MLFloat16>("a_scale", {1}, A_scale);
|
||||
test.AddInput<MLFloat16>("b_scale", {1}, B_scale);
|
||||
test.AddInput<int8_t>("a_zero_point", {1}, A_zero_point);
|
||||
test.AddInput<uint8_t>("b_zero_point", {1}, B_zero_point);
|
||||
test.AddInput<MLFloat16>("bias", {N}, Bias);
|
||||
|
||||
std::vector<float> Y_data(M * N);
|
||||
CalculateMatMulIntegerToFloat<int8_t, uint8_t, MLFloat16>(M, N, K, A_data, A_scale, A_zero_point,
|
||||
B_data, B_scale, B_zero_point, Bias, Y_data,
|
||||
false, true, true);
|
||||
|
||||
test.AddOutput<MLFloat16>("Y", {M, N}, ToFloat16(Y_data));
|
||||
|
||||
std::vector<std::unique_ptr<IExecutionProvider>> execution_providers;
|
||||
execution_providers.push_back(DefaultDmlExecutionProvider());
|
||||
test.Run(OpTester::ExpectResult::kExpectSuccess, "", {}, nullptr, &execution_providers);
|
||||
}
|
||||
|
||||
TEST(MatMulIntegerToFloat, MatMulIntegerToFloat_FP16) {
|
||||
OpTester test("MatMulIntegerToFloat", 1, kMSDomain);
|
||||
int64_t M = 2;
|
||||
int64_t N = 2;
|
||||
int64_t K = 3;
|
||||
|
||||
std::vector<int8_t> A_data = {11, -2, 5,
|
||||
-1, 3, 10};
|
||||
std::vector<int8_t> B_data = {-13, -2,
|
||||
9, 55,
|
||||
-1, 23};
|
||||
std::vector<MLFloat16> A_scale = ToFloat16({0.910f});
|
||||
std::vector<MLFloat16> B_scale = ToFloat16({1.10f, 1.123f});
|
||||
|
||||
std::vector<int8_t> A_zero_point = {113};
|
||||
std::vector<int8_t> B_zero_point = {98, 71};
|
||||
|
||||
std::vector<MLFloat16> Bias = ToFloat16({0.10f, 1.123f});
|
||||
|
||||
test.AddInput<int8_t>("A", {M, K}, A_data);
|
||||
test.AddInput<int8_t>("B", {K, N}, B_data);
|
||||
|
||||
test.AddInput<MLFloat16>("a_scale", {}, {A_scale});
|
||||
test.AddInput<MLFloat16>("b_scale", {N}, B_scale);
|
||||
test.AddInput<int8_t>("a_zero_point", {}, {A_zero_point});
|
||||
test.AddInput<int8_t>("b_zero_point", {N}, B_zero_point);
|
||||
test.AddInput<MLFloat16>("bias", {N}, Bias);
|
||||
|
||||
std::vector<float> Y_data(M * N);
|
||||
CalculateMatMulIntegerToFloat<int8_t, int8_t, MLFloat16>(M, N, K, A_data, A_scale, A_zero_point,
|
||||
B_data, B_scale, B_zero_point, Bias, Y_data,
|
||||
true, true, true);
|
||||
|
||||
test.AddOutput<MLFloat16>("Y", {M, N}, ToFloat16(Y_data));
|
||||
test.SetOutputRelErr("Y", 2e-2f);
|
||||
std::vector<std::unique_ptr<IExecutionProvider>> execution_providers;
|
||||
execution_providers.push_back(DefaultDmlExecutionProvider());
|
||||
test.Run(OpTester::ExpectResult::kExpectSuccess, "", {}, nullptr, &execution_providers);
|
||||
}
|
||||
#endif
|
||||
|
||||
TEST(MatMulIntegerToFloat, MatMulInteger_With_ZeroPoint) {
|
||||
auto test_case = [&](const std::vector<int64_t>& input_shape,
|
||||
const std::vector<int64_t>& weights_shape,
|
||||
|
|
|
|||
|
|
@ -5679,6 +5679,24 @@ TEST_F(GraphTransformationTests, MatMulIntegerToFloatTest) {
|
|||
EXPECT_EQ(op_to_count["Add"], 1);
|
||||
}
|
||||
|
||||
#ifdef USE_DML
|
||||
TEST_F(GraphTransformationTests, MatMulIntegerToFloat16Test) {
|
||||
constexpr const ORTCHAR_T* model_uri = MODEL_FOLDER "fusion/matmul_integer_to_float16_int8.onnx";
|
||||
std::shared_ptr<Model> p_model;
|
||||
ASSERT_STATUS_OK(Model::Load(model_uri, p_model, nullptr, *logger_));
|
||||
Graph& graph = p_model->MainGraph();
|
||||
|
||||
for (auto& node : graph.Nodes()) {
|
||||
node.SetExecutionProviderType(kDmlExecutionProvider);
|
||||
}
|
||||
onnxruntime::GraphTransformerManager graph_transformation_mgr{5};
|
||||
ASSERT_STATUS_OK(graph_transformation_mgr.Register(std::make_unique<MatMulIntegerToFloatFusion>(), TransformerLevel::Level2));
|
||||
ASSERT_STATUS_OK(graph_transformation_mgr.ApplyTransformers(graph, TransformerLevel::Level2, *logger_));
|
||||
std::map<std::string, int> op_to_count = CountOpsInGraph(graph);
|
||||
EXPECT_EQ(op_to_count["com.microsoft.MatMulIntegerToFloat"], 1);
|
||||
}
|
||||
#endif // USE_DML
|
||||
|
||||
#endif
|
||||
|
||||
#ifndef DISABLE_CONTRIB_OPS
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ import onnx
|
|||
from onnx import TensorProto, helper
|
||||
|
||||
|
||||
def GenerateModel(model_name, sign_i, sign_w, has_zp=True, bias=False): # noqa: N802
|
||||
def GenerateModel(model_name, sign_i, sign_w, output_type_fp16, has_zp=True, bias=False): # noqa: N802
|
||||
nodes = [ # subgraph
|
||||
helper.make_node(
|
||||
"MatMulInteger",
|
||||
|
|
@ -13,7 +13,13 @@ def GenerateModel(model_name, sign_i, sign_w, has_zp=True, bias=False): # noqa:
|
|||
"MatMulInteger",
|
||||
),
|
||||
helper.make_node("Mul", ["a_scale", "b_scale"], ["multiplier"], "mul_right"),
|
||||
helper.make_node("Cast", ["matmul_output_int32"], ["matmul_output_float"], "cast", to=1),
|
||||
helper.make_node(
|
||||
"Cast",
|
||||
["matmul_output_int32"],
|
||||
["matmul_output_float"],
|
||||
"cast",
|
||||
to=TensorProto.FLOAT16 if output_type_fp16 else TensorProto.FLOAT,
|
||||
),
|
||||
helper.make_node(
|
||||
"Mul",
|
||||
["matmul_output_float", "multiplier"],
|
||||
|
|
@ -25,8 +31,8 @@ def GenerateModel(model_name, sign_i, sign_w, has_zp=True, bias=False): # noqa:
|
|||
inputs = [ # inputs
|
||||
helper.make_tensor_value_info("A", TensorProto.INT8 if sign_i else TensorProto.UINT8, ["M", "K"]),
|
||||
helper.make_tensor_value_info("B", TensorProto.INT8 if sign_w else TensorProto.UINT8, ["K", "N"]),
|
||||
helper.make_tensor_value_info("a_scale", TensorProto.FLOAT, [1]),
|
||||
helper.make_tensor_value_info("b_scale", TensorProto.FLOAT, ["C"]),
|
||||
helper.make_tensor_value_info("a_scale", TensorProto.FLOAT16 if output_type_fp16 else TensorProto.FLOAT, [1]),
|
||||
helper.make_tensor_value_info("b_scale", TensorProto.FLOAT16 if output_type_fp16 else TensorProto.FLOAT, ["C"]),
|
||||
]
|
||||
|
||||
if has_zp:
|
||||
|
|
@ -48,14 +54,22 @@ def GenerateModel(model_name, sign_i, sign_w, has_zp=True, bias=False): # noqa:
|
|||
if bias:
|
||||
nodes.extend([helper.make_node("Add", ["mul_bottom_output", "bias"], ["Y"], "add")])
|
||||
|
||||
inputs.extend([helper.make_tensor_value_info("bias", TensorProto.FLOAT, ["N"])])
|
||||
inputs.extend(
|
||||
[
|
||||
helper.make_tensor_value_info(
|
||||
"bias", TensorProto.FLOAT16 if output_type_fp16 else TensorProto.FLOAT, ["N"]
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
graph = helper.make_graph(
|
||||
nodes,
|
||||
"DynamicQuantizeMatMul_fusion", # name
|
||||
inputs,
|
||||
[ # outputs
|
||||
helper.make_tensor_value_info("Y", TensorProto.FLOAT, ["M", "N"]),
|
||||
helper.make_tensor_value_info(
|
||||
"Y", TensorProto.FLOAT16 if output_type_fp16 else TensorProto.FLOAT, ["M", "N"]
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
|
|
@ -64,10 +78,32 @@ def GenerateModel(model_name, sign_i, sign_w, has_zp=True, bias=False): # noqa:
|
|||
|
||||
|
||||
if __name__ == "__main__":
|
||||
GenerateModel("matmul_integer_to_float_int8.onnx", False, True)
|
||||
GenerateModel("matmul_integer_to_float_uint8.onnx", False, False)
|
||||
GenerateModel("matmul_integer_to_float_int8_bias.onnx", False, True, False, True)
|
||||
GenerateModel("matmul_integer_to_float_uint8_bias.onnx", False, False, False, True)
|
||||
GenerateModel("matmul_integer_to_float16_int8.onnx", sign_i=False, sign_w=True, output_type_fp16=True)
|
||||
GenerateModel("matmul_integer_to_float_int8.onnx", sign_i=False, sign_w=True, output_type_fp16=False)
|
||||
GenerateModel("matmul_integer_to_float_uint8.onnx", sign_i=False, sign_w=False, output_type_fp16=False)
|
||||
GenerateModel(
|
||||
"matmul_integer_to_float_int8_bias.onnx",
|
||||
sign_i=False,
|
||||
sign_w=True,
|
||||
output_type_fp16=False,
|
||||
has_zp=False,
|
||||
bias=True,
|
||||
)
|
||||
GenerateModel(
|
||||
"matmul_integer_to_float_uint8_bias.onnx",
|
||||
sign_i=False,
|
||||
sign_w=False,
|
||||
output_type_fp16=False,
|
||||
has_zp=False,
|
||||
bias=True,
|
||||
)
|
||||
|
||||
GenerateModel("matmul_integer_to_float_int8_int8.onnx", True, True)
|
||||
GenerateModel("matmul_integer_to_float_int8_int8_bias.onnx", True, True, False, True)
|
||||
GenerateModel("matmul_integer_to_float_int8_int8.onnx", sign_i=True, sign_w=True, output_type_fp16=False)
|
||||
GenerateModel(
|
||||
"matmul_integer_to_float_int8_int8_bias.onnx",
|
||||
sign_i=True,
|
||||
sign_w=True,
|
||||
output_type_fp16=False,
|
||||
has_zp=False,
|
||||
bias=True,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
:ﾌ
|
||||
:ﾌ
|
||||
U
|
||||
A
|
||||
B
|
||||
|
|
@ -44,4 +44,4 @@ mul_bottom"Mul DynamicQuantizeMatMul_fusionZ
|
|||
|
||||
|
||||
M
|
||||
NB
|
||||
NB
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
:ﾄ
|
||||
:ﾄ
|
||||
9
|
||||
A
|
||||
Bmatmul_output_int32
MatMulInteger"
MatMulInteger
|
||||
|
|
@ -41,4 +41,4 @@ mul_bottom"Mul
|
|||
|
||||
|
||||
M
|
||||
NB
|
||||
NB
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
:ﾌ
|
||||
:ﾌ
|
||||
U
|
||||
A
|
||||
B
|
||||
|
|
@ -44,4 +44,4 @@ mul_bottom"Mul DynamicQuantizeMatMul_fusionZ
|
|||
|
||||
|
||||
M
|
||||
NB
|
||||
NB
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
:ﾄ
|
||||
:ﾄ
|
||||
9
|
||||
A
|
||||
Bmatmul_output_int32
MatMulInteger"
MatMulInteger
|
||||
|
|
@ -41,4 +41,4 @@ mul_bottom"Mul
|
|||
|
||||
|
||||
M
|
||||
NB
|
||||
NB
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
:ﾌ
|
||||
:ﾌ
|
||||
U
|
||||
A
|
||||
B
|
||||
|
|
@ -44,4 +44,4 @@ mul_bottom"Mul DynamicQuantizeMatMul_fusionZ
|
|||
|
||||
|
||||
M
|
||||
NB
|
||||
NB
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
:ﾄ
|
||||
:ﾄ
|
||||
9
|
||||
A
|
||||
Bmatmul_output_int32
MatMulInteger"
MatMulInteger
|
||||
|
|
@ -41,4 +41,4 @@ mul_bottom"Mul
|
|||
|
||||
|
||||
M
|
||||
NB
|
||||
NB
|
||||
Binary file not shown.
51
onnxruntime/test/testdata/transform/fusion/matmul_integer_to_float16_int8.onnx
vendored
Normal file
51
onnxruntime/test/testdata/transform/fusion/matmul_integer_to_float16_int8.onnx
vendored
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
:Ě
|
||||
U
|
||||
A
|
||||
B
|
||||
a_zero_point
|
||||
b_zero_pointmatmul_output_int32
MatMulInteger"
MatMulInteger
|
||||
.
|
||||
a_scale
|
||||
b_scale
|
||||
multiplier mul_right"Mul
|
||||
A
|
||||
matmul_output_int32matmul_output_floatcast"Cast*
|
||||
to
|
||||
|
||||
5
|
||||
matmul_output_float
|
||||
|
||||
multiplierY
|
||||
mul_bottom"Mul DynamicQuantizeMatMul_fusionZ
|
||||
A
|
||||
|
||||
|
||||
M
|
||||
KZ
|
||||
B
|
||||
|
||||
|
||||
K
|
||||
NZ
|
||||
a_scale
|
||||
|
||||
|
||||
|
||||
Z
|
||||
b_scale
|
||||
|
||||
|
||||
CZ
|
||||
a_zero_point
|
||||
|
||||
|
||||
Z
|
||||
b_zero_point
|
||||
|
||||
Cb
|
||||
Y
|
||||
|
||||
|
||||
|
||||
M
|
||||
NB
|
||||
Loading…
Reference in a new issue