mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-29 20:14:01 +00:00
QuantizeBFP and DequantizeBFP (#12833)
* `QuantizeBFP` and `DequantizeBFP` schemas - similar to `QuantizeLinear` and `DeQuantizeLinear`. * BFP datatype is represented as a `uint8` tensor with shape and stride metadata. This is preferrable to adding a new datatype for BFP, which is more disruptive and [discouraged by PyTorch](https://discuss.pytorch.org/t/training-with-custom-quantized-datatype/152132/2). Context: The Microsoft Floating Point (BFP) datatype shares an exponent for every n numbers called a “bounding box.” Each number still has its own mantissa and sign bits. BFP has been shown to incur 3-4 less cost (energy and area) than BFloat16 and INT8 counterparts without reductions in accuracy for the ImageNet benchmark as described in [Rouhani 2020](https://proceedings.neurips.cc/paper/2020/file/747e32ab0fea7fbd2ad9ec03daa3f840-Paper.pdf). Requirements: * There are many variants of BFP (number of mantissa bits, number of shared exponent bits, size of bounding box, custom bit fields, etc.) * The size and layout of an BFP variant varies across hardware * bounding box can be over arbitrary dimensions; for example, for the channel "C" dimension in a N x C x H x W tensor for convolution Goals of this PR: * Add initial versions of QuantizeBFP and DequantizeBFP operators to enable QDQ-style quantization with BFP. Once the schemas stabilize, we can consider upstreaming to ONNX. * Add some basic type and shape inferencing tests; tests that run on an EP will be a follow-up.
This commit is contained in:
ashari4
GitHub
parent
057567f39f
commit
c4a7e88fc8
6 changed files with 1174 additions and 1048 deletions
|
|
@ -18,6 +18,7 @@ Do not modify directly.*
|
|||
* <a href="#com.microsoft.ConvTransposeWithDynamicPads">com.microsoft.ConvTransposeWithDynamicPads</a>
|
||||
* <a href="#com.microsoft.CropAndResize">com.microsoft.CropAndResize</a>
|
||||
* <a href="#com.microsoft.DecoderAttention">com.microsoft.DecoderAttention</a>
|
||||
* <a href="#com.microsoft.DequantizeBFP">com.microsoft.DequantizeBFP</a>
|
||||
* <a href="#com.microsoft.DequantizeLinear">com.microsoft.DequantizeLinear</a>
|
||||
* <a href="#com.microsoft.DynamicQuantizeLSTM">com.microsoft.DynamicQuantizeLSTM</a>
|
||||
* <a href="#com.microsoft.DynamicQuantizeMatMul">com.microsoft.DynamicQuantizeMatMul</a>
|
||||
|
|
@ -59,6 +60,7 @@ Do not modify directly.*
|
|||
* <a href="#com.microsoft.QOrderedGelu">com.microsoft.QOrderedGelu</a>
|
||||
* <a href="#com.microsoft.QOrderedLayerNormalization">com.microsoft.QOrderedLayerNormalization</a>
|
||||
* <a href="#com.microsoft.QOrderedMatMul">com.microsoft.QOrderedMatMul</a>
|
||||
* <a href="#com.microsoft.QuantizeBFP">com.microsoft.QuantizeBFP</a>
|
||||
* <a href="#com.microsoft.QuantizeLinear">com.microsoft.QuantizeLinear</a>
|
||||
* <a href="#com.microsoft.Range">com.microsoft.Range</a>
|
||||
* <a href="#com.microsoft.ReduceSumInteger">com.microsoft.ReduceSumInteger</a>
|
||||
|
|
@ -985,6 +987,55 @@ This version of the operator has been available since version 1 of the 'com.micr
|
|||
</dl>
|
||||
|
||||
|
||||
### <a name="com.microsoft.DequantizeBFP"></a><a name="com.microsoft.dequantizebfp">**com.microsoft.DequantizeBFP**</a>
|
||||
|
||||
The BFP dequantization operator. It consumes the raw BFP data and some metadata such as the shape and strides of the original tensor and computes the dequantized tensor.
|
||||
|
||||
#### Version
|
||||
|
||||
This version of the operator has been available since version 1 of the 'com.microsoft' operator set.
|
||||
|
||||
#### Attributes
|
||||
|
||||
<dl>
|
||||
<dt><tt>bfp_type</tt> : int (required)</dt>
|
||||
<dd>The type of BFP - must match with the BFPType enum</dd>
|
||||
<dt><tt>block_dims</tt> : list of ints</dt>
|
||||
<dd>Numbers within a bounding box will span across these dimensions.Any dimension not in this list is the same for all numbers within a bounding box.As an example, consider a 2D tensor with shape [d0, d1] and block_dims equal to [1].Within a bounding box, all elements will be within the same row but will be from different columnns.The default is the last dimension.</dd>
|
||||
<dt><tt>dtype</tt> : int</dt>
|
||||
<dd>The datatype to dequantize to.</dd>
|
||||
</dl>
|
||||
|
||||
#### Inputs
|
||||
|
||||
<dl>
|
||||
<dt><tt>x</tt> : T1</dt>
|
||||
<dd>1-D, contiguous, raw, BFP data to be de-quantized.</dd>
|
||||
<dt><tt>shape</tt> : T2</dt>
|
||||
<dd>shape of the original tensor.</dd>
|
||||
<dt><tt>strides</tt> : T2</dt>
|
||||
<dd>strides of the original tensor.</dd>
|
||||
</dl>
|
||||
|
||||
#### Outputs
|
||||
|
||||
<dl>
|
||||
<dt><tt>y</tt> : T3</dt>
|
||||
<dd>de-quantized tensor.</dd>
|
||||
</dl>
|
||||
|
||||
#### Type Constraints
|
||||
|
||||
<dl>
|
||||
<dt><tt>T1</tt> : tensor(uint8)</dt>
|
||||
<dd>Constrain the input to uint8.</dd>
|
||||
<dt><tt>T2</tt> : tensor(int64)</dt>
|
||||
<dd>Constrain shape and strides to uint64.</dd>
|
||||
<dt><tt>T3</tt> : tensor(float), tensor(float16), tensor(bfloat16)</dt>
|
||||
<dd>Constrain y to float and bfloat16.</dd>
|
||||
</dl>
|
||||
|
||||
|
||||
### <a name="com.microsoft.DequantizeLinear"></a><a name="com.microsoft.dequantizelinear">**com.microsoft.DequantizeLinear**</a>
|
||||
|
||||
The linear dequantization operator. It consumes a quantized data, a scale, a zero point and computes the full precision data.
|
||||
|
|
@ -3022,6 +3073,53 @@ This version of the operator has been available since version 1 of the 'com.micr
|
|||
</dl>
|
||||
|
||||
|
||||
### <a name="com.microsoft.QuantizeBFP"></a><a name="com.microsoft.quantizebfp">**com.microsoft.QuantizeBFP**</a>
|
||||
|
||||
The BFP quantization operator. It consumes a full precision tensor and computes an BFP tensor.
|
||||
|
||||
#### Version
|
||||
|
||||
This version of the operator has been available since version 1 of the 'com.microsoft' operator set.
|
||||
|
||||
#### Attributes
|
||||
|
||||
<dl>
|
||||
<dt><tt>bfp_type</tt> : int (required)</dt>
|
||||
<dd>The type of BFP - must match with the BFPType enum</dd>
|
||||
<dt><tt>block_dims</tt> : list of ints</dt>
|
||||
<dd>Numbers within a bounding box will span across these dimensions.Any dimension not in this list is the same for all numbers within a bounding box.As an example, consider a 2D tensor with shape [d0, d1] and block_dims equal to [1].Within a bounding box, all elements will be within the same row but will be from different columnns.The default is the last dimension.</dd>
|
||||
</dl>
|
||||
|
||||
#### Inputs
|
||||
|
||||
<dl>
|
||||
<dt><tt>x</tt> : T1</dt>
|
||||
<dd>N-D full precision input tensor to be quantized.</dd>
|
||||
</dl>
|
||||
|
||||
#### Outputs
|
||||
|
||||
<dl>
|
||||
<dt><tt>y</tt> : T2</dt>
|
||||
<dd>1-D, contiguous BFP data</dd>
|
||||
<dt><tt>shape</tt> : T3</dt>
|
||||
<dd>Shape of x</dd>
|
||||
<dt><tt>strides</tt> : T3</dt>
|
||||
<dd>Strides of x</dd>
|
||||
</dl>
|
||||
|
||||
#### Type Constraints
|
||||
|
||||
<dl>
|
||||
<dt><tt>T1</tt> : tensor(float), tensor(float16), tensor(bfloat16)</dt>
|
||||
<dd>Constrain the input to float and bfloat.</dd>
|
||||
<dt><tt>T2</tt> : tensor(uint8)</dt>
|
||||
<dd>Constrain y to uint8.</dd>
|
||||
<dt><tt>T3</tt> : tensor(int64)</dt>
|
||||
<dd>Constrain shape and strides to uint64.</dd>
|
||||
</dl>
|
||||
|
||||
|
||||
### <a name="com.microsoft.QuantizeLinear"></a><a name="com.microsoft.quantizelinear">**com.microsoft.QuantizeLinear**</a>
|
||||
|
||||
The linear quantization operator. It consumes a full precision data, a scale, a zero point to compute the low precision / quantized tensor.
|
||||
|
|
|
|||
|
|
@ -7,15 +7,16 @@
|
|||
|
||||
namespace onnxruntime {
|
||||
namespace contrib {
|
||||
//NHWC ops
|
||||
// NHWC ops
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, NhwcMaxPool);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QLinearGlobalAveragePool);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QLinearAveragePool);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QLinearConv);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, NhwcConv);
|
||||
|
||||
//Quantization ops
|
||||
// Quantization ops
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, DequantizeLinear);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, DequantizeBFP);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, DynamicQuantizeLSTM);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, DynamicQuantizeMatMul);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, MatMulIntegerToFloat);
|
||||
|
|
@ -31,12 +32,13 @@ class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QLinearReduceMean);
|
|||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QLinearSigmoid);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QLinearSoftmax);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QuantizeLinear);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QuantizeBFP);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, ReduceSumInteger);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QOrderedMatMul);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QOrderedGelu);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QOrderedLayerNormalization);
|
||||
|
||||
//Others
|
||||
// Others
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, Attention);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, BeamSearch);
|
||||
class ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, BiasDropout);
|
||||
|
|
@ -91,6 +93,7 @@ class OpSet_Microsoft_ver1 {
|
|||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, NhwcConv)>());
|
||||
|
||||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, DequantizeLinear)>());
|
||||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, DequantizeBFP)>());
|
||||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, DynamicQuantizeLSTM)>());
|
||||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, DynamicQuantizeMatMul)>());
|
||||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, MatMulIntegerToFloat)>());
|
||||
|
|
@ -104,6 +107,7 @@ class OpSet_Microsoft_ver1 {
|
|||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QLinearSigmoid)>());
|
||||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QLinearSoftmax)>());
|
||||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QuantizeLinear)>());
|
||||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QuantizeBFP)>());
|
||||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, ReduceSumInteger)>());
|
||||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QOrderedMatMul)>());
|
||||
fn(GetOpSchema<ONNX_OPERATOR_SET_SCHEMA_CLASS_NAME(Microsoft, 1, QOrderedGelu)>());
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -2,6 +2,8 @@
|
|||
// Licensed under the MIT License.
|
||||
#pragma once
|
||||
|
||||
#include <stdexcept>
|
||||
|
||||
#if !defined(ORT_MINIMAL_BUILD)
|
||||
#include "onnx/defs/schema.h"
|
||||
#else
|
||||
|
|
@ -13,12 +15,75 @@
|
|||
namespace onnxruntime {
|
||||
namespace contrib {
|
||||
|
||||
void ValidateTypeAndShapeForScaleAndZP(
|
||||
ONNX_NAMESPACE::InferenceContext& ctx,
|
||||
int index,
|
||||
::google::protobuf::int32 expectedType,
|
||||
bool isScalar,
|
||||
int expectedTensorSize = 0);
|
||||
void ValidateTypeAndShapeForScaleAndZP(ONNX_NAMESPACE::InferenceContext& ctx, int index,
|
||||
::google::protobuf::int32 expectedType, bool isScalar,
|
||||
int expectedTensorSize = 0);
|
||||
|
||||
// Names follow the convention of BFP_<# sign bits>_<# mantissa bits>_<# exponent bits>_<size of bounding box>
|
||||
enum class BFPType : int64_t {
|
||||
// 1 sign bit, 8 mantissa bits, 8 exponent bits, 16 numbers per bounding box.
|
||||
BFP_1_8_8_16,
|
||||
|
||||
// 1 sign bit, 8 mantissa bits, 8 exponent bits, 16 numbers per bounding box.
|
||||
BFP_1_4_8_16,
|
||||
|
||||
// No sign bit, 8 mantissa bits, 8 exponent bits, 64 numbers per bounding box. Mantissa and exponent are signed.
|
||||
BFP_0_8_8_64,
|
||||
|
||||
// No sign bit, 8 mantissa bits, 8 exponent bits, 128 numbers per bounding box. Mantissa and exponent are signed.
|
||||
BFP_0_8_8_128,
|
||||
|
||||
// No sign bit, 8 mantissa bits, 8 exponent bits, 256 numbers per bounding box. Mantissa and exponent are signed.
|
||||
BFP_0_8_8_256,
|
||||
|
||||
// No sign bit, 4 mantissa bits, 8 exponent bits, 128 numbers per bounding box. Mantissa and exponent are signed.
|
||||
BFP_0_4_8_128,
|
||||
|
||||
// No sign bit, 4 mantissa bits, 8 exponent bits, 256 numbers per bounding box. Mantissa and exponent are signed.
|
||||
BFP_0_4_8_256,
|
||||
|
||||
// No sign bit, 16 mantissa bits, 8 exponent bits, 64 numbers per bounding box. Mantissa and exponent are signed.
|
||||
BFP_0_16_8_64,
|
||||
|
||||
// No sign bit, 16 mantissa bits, 8 exponent bits, 128 numbers per bounding box. Mantissa and exponent are signed.
|
||||
BFP_0_16_8_128,
|
||||
|
||||
// No sign bit, 16 mantissa bits, 8 exponent bits, 256 numbers per bounding box. Mantissa and exponent are signed.
|
||||
BFP_0_16_8_256,
|
||||
|
||||
// No sign bit, 16 mantissa bits, 8 exponent bits, 1 number per bounding box. Mantissa and exponent are signed.
|
||||
BFP_0_16_8_1,
|
||||
|
||||
// No sign bit, 24 mantissa bits, 8 exponent bits, 1 number per bounding box. Mantissa and exponent are signed.
|
||||
BFP_0_24_8_1,
|
||||
|
||||
// Reserved for custom BFP types
|
||||
Custom_BFP_0,
|
||||
Custom_BFP_1
|
||||
};
|
||||
|
||||
inline size_t get_bounding_box_size(const BFPType& bfp_type) {
|
||||
switch (bfp_type) {
|
||||
case BFPType::BFP_0_16_8_1:
|
||||
case BFPType::BFP_0_24_8_1:
|
||||
return 1u;
|
||||
case BFPType::BFP_1_4_8_16:
|
||||
case BFPType::BFP_1_8_8_16:
|
||||
return 16u;
|
||||
case BFPType::BFP_0_8_8_64:
|
||||
case BFPType::BFP_0_16_8_64:
|
||||
return 64u;
|
||||
case BFPType::BFP_0_4_8_128:
|
||||
case BFPType::BFP_0_8_8_128:
|
||||
case BFPType::BFP_0_16_8_128:
|
||||
return 128u;
|
||||
case BFPType::BFP_0_4_8_256:
|
||||
case BFPType::BFP_0_8_8_256:
|
||||
case BFPType::BFP_0_16_8_256:
|
||||
return 256u;
|
||||
default:
|
||||
ONNX_THROW_EX(std::invalid_argument("Unsuported bfp_type case."));
|
||||
}
|
||||
}
|
||||
} // namespace onnxruntime
|
||||
} // namespace contrib
|
||||
} // namespace onnxruntime
|
||||
|
|
|
|||
118
onnxruntime/test/contrib_ops/quantize_bfp_test.cc
Normal file
118
onnxruntime/test/contrib_ops/quantize_bfp_test.cc
Normal file
|
|
@ -0,0 +1,118 @@
|
|||
// Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
// Licensed under the MIT License.
|
||||
|
||||
#include "gtest/gtest.h"
|
||||
#include "test/providers/provider_test_utils.h"
|
||||
#include "core/graph/contrib_ops/quantization_defs.h"
|
||||
#include "core/graph/contrib_ops/contrib_defs.h"
|
||||
#include "core/graph/contrib_ops/ms_opset.h"
|
||||
|
||||
namespace onnxruntime {
|
||||
namespace test {
|
||||
|
||||
TEST(QuantizeBFPTest, CreateQuantizeGraph) {
|
||||
std::unordered_map<std::string, int> domain_to_version;
|
||||
domain_to_version[onnxruntime::kMSDomain] = 1;
|
||||
// Generate the input & output def lists
|
||||
std::vector<ONNX_NAMESPACE::FunctionProto> model_specific_functions;
|
||||
auto p_model = std::make_unique<Model>(
|
||||
"test", true, ModelMetaData(), PathString(), IOnnxRuntimeOpSchemaRegistryList(), domain_to_version,
|
||||
model_specific_functions, DefaultLoggingManager().DefaultLogger(), ModelOptions(true, true));
|
||||
onnxruntime::Graph& graph = p_model->MainGraph();
|
||||
|
||||
ONNX_NAMESPACE::TypeProto x_float;
|
||||
x_float.mutable_tensor_type()->set_elem_type(ONNX_NAMESPACE::TensorProto_DataType_FLOAT);
|
||||
x_float.mutable_tensor_type()->mutable_shape()->add_dim()->set_dim_value(128);
|
||||
x_float.mutable_tensor_type()->mutable_shape()->add_dim()->set_dim_value(16);
|
||||
std::vector<onnxruntime::NodeArg*> input_defs;
|
||||
auto& input_arg_x = graph.GetOrCreateNodeArg("x", &x_float);
|
||||
input_defs.push_back(&input_arg_x);
|
||||
|
||||
NodeAttributes attributes;
|
||||
ONNX_NAMESPACE::AttributeProto bfp_type;
|
||||
bfp_type.set_name("bfp_type");
|
||||
bfp_type.set_i(static_cast<int64_t>(onnxruntime::contrib::BFPType::BFP_1_8_8_16));
|
||||
bfp_type.set_type(ONNX_NAMESPACE::AttributeProto_AttributeType::AttributeProto_AttributeType_INT);
|
||||
attributes["bfp_type"] = bfp_type;
|
||||
ONNX_NAMESPACE::AttributeProto block_dims;
|
||||
block_dims.set_name("block_dims");
|
||||
block_dims.add_ints(1); // bounding box is over dimension 1
|
||||
block_dims.set_type(ONNX_NAMESPACE::AttributeProto_AttributeType::AttributeProto_AttributeType_INTS);
|
||||
attributes["block_dims"] = block_dims;
|
||||
|
||||
std::vector<onnxruntime::NodeArg*> output_defs;
|
||||
ONNX_NAMESPACE::TypeProto y_byte;
|
||||
y_byte.mutable_tensor_type()->set_elem_type(ONNX_NAMESPACE::TensorProto_DataType_UINT8);
|
||||
auto& output_arg_y = graph.GetOrCreateNodeArg("y", &y_byte);
|
||||
output_defs.push_back(&output_arg_y);
|
||||
|
||||
ONNX_NAMESPACE::TypeProto tensor_int;
|
||||
tensor_int.mutable_tensor_type()->set_elem_type(ONNX_NAMESPACE::TensorProto_DataType_INT64);
|
||||
auto& output_arg_shape = graph.GetOrCreateNodeArg("shape", &tensor_int);
|
||||
auto& output_arg_strides = graph.GetOrCreateNodeArg("strides", &tensor_int);
|
||||
output_defs.push_back(&output_arg_shape);
|
||||
output_defs.push_back(&output_arg_strides);
|
||||
|
||||
// Create a simple model
|
||||
graph.AddNode("node1", "QuantizeBFP", "quantizes float tensor to BFP", input_defs, output_defs, &attributes,
|
||||
onnxruntime::kMSDomain);
|
||||
Status status = graph.Resolve();
|
||||
ASSERT_TRUE(status.IsOK()) << status.ErrorMessage();
|
||||
}
|
||||
|
||||
TEST(DequantizeBFPTest, CreateDequantizeGraph) {
|
||||
std::unordered_map<std::string, int> domain_to_version;
|
||||
domain_to_version[onnxruntime::kMSDomain] = 1;
|
||||
// Generate the input & output def lists
|
||||
std::vector<ONNX_NAMESPACE::FunctionProto> model_specific_functions;
|
||||
auto p_model = std::make_unique<Model>(
|
||||
"test", true, ModelMetaData(), PathString(), IOnnxRuntimeOpSchemaRegistryList(), domain_to_version,
|
||||
model_specific_functions, DefaultLoggingManager().DefaultLogger(), ModelOptions(true, true));
|
||||
onnxruntime::Graph& graph = p_model->MainGraph();
|
||||
|
||||
ONNX_NAMESPACE::TypeProto x_byte;
|
||||
x_byte.mutable_tensor_type()->set_elem_type(ONNX_NAMESPACE::TensorProto_DataType_UINT8);
|
||||
x_byte.mutable_tensor_type()->mutable_shape()->add_dim()->set_dim_value(128); // an arbitrary byte size
|
||||
std::vector<onnxruntime::NodeArg*> input_defs;
|
||||
auto& input_arg_x = graph.GetOrCreateNodeArg("x", &x_byte);
|
||||
input_defs.push_back(&input_arg_x);
|
||||
|
||||
ONNX_NAMESPACE::TypeProto tensor_int;
|
||||
tensor_int.mutable_tensor_type()->set_elem_type(ONNX_NAMESPACE::TensorProto_DataType_INT64);
|
||||
tensor_int.mutable_tensor_type()->mutable_shape()->add_dim()->set_dim_value(2);
|
||||
auto& input_arg_shape = graph.GetOrCreateNodeArg("shape", &tensor_int);
|
||||
auto& input_arg_strides = graph.GetOrCreateNodeArg("strides", &tensor_int);
|
||||
input_defs.push_back(&input_arg_shape);
|
||||
input_defs.push_back(&input_arg_strides);
|
||||
|
||||
NodeAttributes attributes;
|
||||
ONNX_NAMESPACE::AttributeProto bfp_type;
|
||||
bfp_type.set_name("bfp_type");
|
||||
bfp_type.set_i(static_cast<int64_t>(onnxruntime::contrib::BFPType::BFP_1_8_8_16));
|
||||
bfp_type.set_type(ONNX_NAMESPACE::AttributeProto_AttributeType::AttributeProto_AttributeType_INT);
|
||||
attributes["bfp_type"] = bfp_type;
|
||||
ONNX_NAMESPACE::AttributeProto block_dims;
|
||||
block_dims.set_name("block_dims");
|
||||
block_dims.add_ints(1); // bounding box is over dimension 1
|
||||
block_dims.set_type(ONNX_NAMESPACE::AttributeProto_AttributeType::AttributeProto_AttributeType_INTS);
|
||||
attributes["block_dims"] = block_dims;
|
||||
ONNX_NAMESPACE::AttributeProto dtype;
|
||||
dtype.set_name("dtype");
|
||||
dtype.set_i(static_cast<int64_t>(ONNX_NAMESPACE::TensorProto_DataType_FLOAT));
|
||||
dtype.set_type(ONNX_NAMESPACE::AttributeProto_AttributeType::AttributeProto_AttributeType_INT);
|
||||
attributes["dtype"] = dtype;
|
||||
|
||||
std::vector<onnxruntime::NodeArg*> output_defs;
|
||||
ONNX_NAMESPACE::TypeProto y_float;
|
||||
y_float.mutable_tensor_type()->set_elem_type(ONNX_NAMESPACE::TensorProto_DataType_FLOAT);
|
||||
auto& output_arg_y = graph.GetOrCreateNodeArg("y", &y_float);
|
||||
output_defs.push_back(&output_arg_y);
|
||||
|
||||
// Create a simple model
|
||||
graph.AddNode("node1", "DequantizeBFP", "dequantizes BFP tensor to float", input_defs, output_defs, &attributes,
|
||||
onnxruntime::kMSDomain);
|
||||
Status status = graph.Resolve();
|
||||
ASSERT_TRUE(status.IsOK()) << status.ErrorMessage();
|
||||
}
|
||||
} // namespace test
|
||||
} // namespace onnxruntime
|
||||
|
|
@ -238,7 +238,6 @@ class AdasumInterface {
|
|||
tensor_counts[i] = tensor_counts[i];
|
||||
nghrCountVec[nghrCountVec_index][i] = 0;
|
||||
} else {
|
||||
assert((myCount - myCountSoFar) >= 0);
|
||||
nghrCountVec[nghrCountVec_index][i] =
|
||||
tensor_counts[i] -
|
||||
(myCount - myCountSoFar); // should not be negative
|
||||
|
|
|
|||
Loading…
Reference in a new issue