diff --git a/docs/ContribOperators.md b/docs/ContribOperators.md index 2290030073..692d00f3f4 100644 --- a/docs/ContribOperators.md +++ b/docs/ContribOperators.md @@ -115,33 +115,31 @@ Do not modify directly.* * com.microsoft.WordConvEmbedding * experimental com.microsoft.IsAllFinite * experimental com.microsoft.QEmbedLayerNormalization +* com.microsoft.nchwc + * com.microsoft.nchwc.AveragePool + * com.microsoft.nchwc.Conv + * com.microsoft.nchwc.GlobalAveragePool + * com.microsoft.nchwc.GlobalMaxPool + * com.microsoft.nchwc.MaxPool + * com.microsoft.nchwc.ReorderInput + * com.microsoft.nchwc.ReorderOutput + * com.microsoft.nchwc.Upsample +* com.ms.internal.nhwc + * com.ms.internal.nhwc.BatchNormalization + * com.ms.internal.nhwc.ConvTranspose + * com.ms.internal.nhwc.DepthToSpace + * com.ms.internal.nhwc.GlobalLpPool + * com.ms.internal.nhwc.InstanceNormalization + * com.ms.internal.nhwc.LRN + * com.ms.internal.nhwc.LpPool + * com.ms.internal.nhwc.MaxUnpool + * com.ms.internal.nhwc.QLinearConvTranspose + * com.ms.internal.nhwc.Resize + * com.ms.internal.nhwc.SpaceToDepth ## com.microsoft ### **com.microsoft.Attention** - Multi-Head Attention that can be either unidirectional (like GPT-2) or bidirectional (like BERT). - - The weights for input projection of Q, K and V are merged. The data is stacked on the second dimension. Its shape - is (input_hidden_size, hidden_size + hidden_size + v_hidden_size). Here hidden_size is the hidden dimension of Q and K, - and v_hidden_size is that of V. - - The mask_index is optional. Besides raw attention mask with shape (batch_size, total_sequence_length) - or (batch_size, sequence_length, total_sequence_length) with value 0 for masked and 1 otherwise, - we support other two formats: When input has right-side padding, mask_index is one dimension with shape (batch_size), - where value is actual sequence length excluding padding. When input has left-side padding, mask_index has - shape (2 * batch_size), where the values are the exclusive end positions followed by the inclusive start positions. - - When unidirectional is 1, each token only attends to previous tokens. - - Both past and present state are optional. They shall be used together, and not allowed to use only one of them. - The qkv_hidden_sizes is required only when K and V have different hidden sizes. - - When there is past state, hidden dimension for Q, K and V shall be the same. - - The total_sequence_length is past_sequence_length + kv_sequence_length. Here kv_sequence_length is the length of K or V. - For self attention, kv_sequence_length equals to sequence_length (sequence length of Q). - For cross attention, query and key might have different lengths. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -171,28 +169,28 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T
-
Input tensor with shape (batch_size, sequence_length, input_hidden_size)
+
weights : T
-
Merged Q/K/V weights with shape (input_hidden_size, hidden_size + hidden_size + v_hidden_size)
+
bias (optional) : T
-
Bias tensor with shape (hidden_size + hidden_size + v_hidden_size) for input projection
+
mask_index (optional) : M
-
Attention mask with shape (batch_size, 1, max_sequence_length, max_sequence_length), (batch_size, total_sequence_length) or (batch_size, sequence_length, total_sequence_length), or index with shape (batch_size) or (2 * batch_size) or (3 * batch_size + 2)
+
past (optional) : T
-
past state for key and value with shape (2, batch_size, num_heads, past_sequence_length, head_size)When past_present_share_buffer is set, its shape is (2, batch_size, num_heads, max_sequence_length, head_size)
+
attention_bias (optional) : T
-
additional add to QxK' with shape (batch_size or 1, num_heads or 1, sequence_length, total_sequence_length)
+
past_sequence_length (optional) : M
-
When past_present_share_buffer is used, it is required to specify past_sequence_length (could be 0).
+
#### Outputs (1 - 2)
output : T
-
3D output tensor with shape (batch_size, sequence_length, v_hidden_size)
+
present (optional) : T
-
past state for key and value with shape (2, batch_size, num_heads, total_sequence_length, head_size). If past_present_share_buffer is set, its shape is (2, batch_size, num_heads, max_sequence_length, head_size), while effective_seq_length = (past_sequence_length + kv_sequence_length).
+
#### Type Constraints @@ -207,136 +205,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.AttnLSTM** - Computes an one-layer RNN where its RNN Cell is an AttentionWrapper wrapped a LSTM Cell. The RNN layer - contains following basic component: LSTM Cell, Bahdanau Attention Mechanism, AttentionWrapp. - - Activation functions: - - Relu(x) - max(0, x) - - Tanh(x) - (1 - e^{-2x})/(1 + e^{-2x}) - - Sigmoid(x) - 1/(1 + e^{-x}) - - (NOTE: Below are optional) - - Affine(x) - alpha*x + beta - - LeakyRelu(x) - x if x >= 0 else alpha * x - - ThresholdedRelu(x) - x if x >= alpha else 0 - - ScaledTanh(x) - alpha*Tanh(beta*x) - - HardSigmoid(x) - min(max(alpha*x + beta, 0), 1) - - Elu(x) - x if x >= 0 else alpha*(e^x - 1) - - Softsign(x) - x/(1 + |x|) - - Softplus(x) - log(1 + e^x) - - Softmax(x) - exp(x) / sum(exp(x)) - - Bahdanau Attention Mechanism: - `M` - Memory tensor. - - `VALUES` - masked Memory by its real sequence length. - - `MW` - Memory layer weight. - - `KEYS` - Processed memory tensor by the memory layer. - KEYS = M * MW - - `Query` - Query tensor, normally at specific time step in sequence. - - `QW` - Query layer weight in the attention mechanism - - `PQ` - processed query, = `Query` * `QW` - - `V' - attention vector - - `ALIGN` - calculated alignment based on Query and KEYS - ALIGN = softmax(reduce_sum(`V` * Tanh(`KEYS` + `PQ`))) - - `CONTEXT` - context based on `ALIGN` and `VALUES` - CONTEXT = `ALIGN` * `VALUES` - - - LSTM Cell: - `X` - input tensor concat with attention state in the attention wrapper - - `i` - input gate - - `o` - output gate - - `f` - forget gate - - `c` - cell gate - - `t` - time step (t-1 means previous time step) - - `W[iofc]` - W parameter weight matrix for input, output, forget, and cell gates - - `R[iofc]` - R recurrence weight matrix for input, output, forget, and cell gates - - `Wb[iofc]` - W bias vectors for input, output, forget, and cell gates - - `Rb[iofc]` - R bias vectors for input, output, forget, and cell gates - - `P[iof]` - P peephole weight vector for input, output, and forget gates - - `WB[iofc]` - W parameter weight matrix for backward input, output, forget, and cell gates - - `RB[iofc]` - R recurrence weight matrix for backward input, output, forget, and cell gates - - `WBb[iofc]` - W bias vectors for backward input, output, forget, and cell gates - - `RBb[iofc]` - R bias vectors for backward input, output, forget, and cell gates - - `PB[iof]` - P peephole weight vector for backward input, output, and forget gates - - `H` - Hidden state - - `num_directions` - 2 if direction == bidirectional else 1 - - Equations (Default: f=Sigmoid, g=Tanh, h=Tanh): - - - it = f(Xt*(Wi^T) + Ht-1*(Ri^T) + Pi (.) Ct-1 + Wbi + Rbi) - - - ft = f(Xt*(Wf^T) + Ht-1*(Rf^T) + Pf (.) Ct-1 + Wbf + Rbf) - - - ct = g(Xt*(Wc^T) + Ht-1*(Rc^T) + Wbc + Rbc) - - - Ct = ft (.) Ct-1 + it (.) ct - - - ot = f(Xt*(Wo^T) + Ht-1*(Ro^T) + Po (.) Ct + Wbo + Rbo) - - - Ht = ot (.) h(Ct) - - - AttentionWrapp Notations: - `lstm()' - wrapped inner cell. - Ht, Ct = lstm(concat(Xt, ATTNt-1), Ct-1) - - `am()` - attention mechanism the wrapper used. - CONTEXTt, ALIGNt = am(Ht, ALIGNt-1) - - `AW` - attention layer weights, optional. - - `ATTN` - attention state, initial is zero. If `AW` provided, it is the output of the attention layer, - ATTNt = concat(Ht, CONTEXTt) * AW - otherwise, - ATTNt = CONTEXTt - - RNN layer output: - `Y` - if needed is the sequence of Ht from lstm cell. - - `Y_h` - is the last valid H from lstm cell. - - `Y_c` - is the last valid C from lstm cell. - - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -364,44 +232,44 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
The input sequences packed (and potentially padded) into one 3-D tensor with the shape of `[seq_length, batch_size, input_size]`
+
W : T
-
The weight tensor for the gates. Concatenation of `W[iofc]` and `WB[iofc]` (if bidirectional) along dimension 0. The tensor has shape `[num_directions, 4*hidden_size, input_size]`.
+
R : T
-
The recurrence weight tensor. Concatenation of `R[iofc]` and `RB[iofc]` (if bidirectional) along dimension 0. This tensor has shape `[num_directions, 4*hidden_size, hidden_size]`.
+
B (optional) : T
-
The bias tensor for input gate. Concatenation of `[Wb[iofc], Rb[iofc]]`, and `[WBb[iofc], RBb[iofc]]` (if bidirectional) along dimension 0. This tensor has shape `[num_directions, 8*hidden_size]`. Optional: If not specified - assumed to be 0.
+
sequence_lens (optional) : T1
-
Optional tensor specifying lengths of the sequences in a batch. If not specified - assumed all sequences in the batch to have length `seq_length`. It has shape `[batch_size]`
+
initial_h (optional) : T
-
Optional initial value of the hidden. If not specified - assumed to be 0. It has shape `[num_directions, batch_size, hidden_size]`.
+
initial_c (optional) : T
-
Optional initial value of the cell. If not specified - assumed to be 0. It has shape `[num_directions, batch_size, hidden_size]`.
+
P (optional) : T
-
The weight tensor for peepholes. Concatenation of `P[iof]` and `PB[iof]` (if bidirectional) along dimension 0. It has shape `[num_directions, 3*hidde_size]`. Optional: If not specified - assumed to be 0.
+
QW (optional) : T
-
The weight tensor of the query layer in the attention mechanism. Should be of shape `[num_directions, am_query_depth(hidden_size of lstm), am_attn_size]`
+
MW (optional) : T
-
The weight tensor of the memory layer in the attention mechanism. Should be of shape `[num_directions, memory_depth, am_attn_size]`
+
V (optional) : T
-
The attention_v tensor in the attention mechanism. Should be of shape `[num_directions, am_attn_size]`
+
M (optional) : T
-
The sequence of the memory (input) for attention mechanism. Should be of `[batch_size, max_memory_step, memory_depth]`
+
memory_seq_lens (optional) : T1
-
The sequence length of the input memory for the attention mechanism. Should be of `[batch_size]`
+
AW (optional) : T
-
The weights of attention layer in the attention wrapper. If exists, should be of shape `[num_directions, memory_depth+hidden_size, aw_attn_size]. Please note that attention mechanism context depth is also memory_depth in the attention mechanism.`
+
#### Outputs (0 - 3)
Y (optional) : T
-
A tensor that concats all the intermediate output values of the hidden. It has shape `[seq_length, num_directions, batch_size, hidden_size]`
+
Y_h (optional) : T
-
The last output value of the hidden. It has shape `[num_directions, batch_size, hidden_size]`.
+
Y_c (optional) : T
-
The last output value of the cell. It has shape `[num_directions, batch_size, hidden_size]`.
+
#### Type Constraints @@ -416,8 +284,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.BeamSearch** - Beam Search for text generation. Supports GPT-2 decoder. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -451,40 +317,40 @@ This version of the operator has been available since version 1 of the 'com.micr
input_ids : F
-
The sequence used as a prompt for the generation in the encoder subgraph. Shape is (batch_size, sequence_length)
+
max_length : I
-
The maximum length of the sequence to be generated. Shape is (1)
+
min_length (optional) : I
-
The minimum length below which the score of eos_token_id is set to -Inf. Shape is (1)
+
num_beams : I
-
Number of beams for beam search. 1 means no beam search. Shape is (1)
+
num_return_sequences : I
-
The number of returned sequences in the batch. Shape is (1)
+
length_penalty (optional) : T
-
Exponential penalty to the length. Default value 1.0 means no penalty.Value > 1.0 encourages longer sequences, while values < 1.0 produces shorter sequences.Shape is (1,)
+
repetition_penalty (optional) : T
-
The parameter for repetition penalty. Default value 1.0 means no penalty. Accepts value > 0.0. Shape is (1)
+
vocab_mask (optional) : M
-
Mask of vocabulary. Words that masked with 0 are not allowed to be generated, and 1 is allowed. Shape is (vocab_size)
+
prefix_vocab_mask (optional) : M
-
Mask of vocabulary for first step. Words that masked with 0 are not allowed to be generated, and 1 is allowed. Shape is (batch_size, vocab_size)
+
attention_mask (optional) : I
-
Custom attention mask. Shape is (batch_size, sequence_length)
+
decoder_input_ids (optional) : I
-
The forced input id sequence for the decoder subgraph. Shape is (batch_size, initial_sequence_length)
+
logits_processor (optional) : I
-
Specific logits processor for different types of beamsearch models. Default value 0 means no specific logit processor. Accepts value >= 0. Shape is (1)
+
#### Outputs (1 - 3)
sequences : I
-
Word IDs of generated sequences. Shape is (batch_size, num_return_sequences, max_sequence_length)
+
sequences_scores (optional) : T
-
Final beam score of the generated sequences. Shape is (batch_size, num_return_sequences)
+
scores (optional) : T
-
Processed beam scores for each vocabulary token at each generation step.Beam scores consisting of log softmax scores for each vocabulary token and sum of log softmax of previously generated tokens in this beam.Shape is (max_length - sequence_length, batch_size, num_beams, vocab_size)
+
#### Type Constraints @@ -503,8 +369,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.BiasAdd** - Add input with bias, then add residual inputs. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -513,18 +377,18 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
Input tensor. Dimensions are (N, S, C), where N is the batch size, S is image size H*W, and C is number of channels
+
bias : T
-
Bias tensor. Dimensions are (C)
+
skip : T
-
Residual tensor. Dimensions are (N, S, C)
+
#### Outputs
Y : T
-
The output tensor with dimensions (N, S, C)
+
#### Type Constraints @@ -537,8 +401,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.BiasDropout** - output, dropout_mask = Dropout(data + bias, ratio) + residual, Intended to specialize the dropout pattern commonly found in transformer models. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -554,24 +416,24 @@ This version of the operator has been available since version 1 of the 'com.micr
data : T
-
The input data as Tensor.
+
bias : T
-
The bias input, a vector with the same shape as last dim of data OR same shape with data
+
residual (optional) : T
-
The residual input, must have the same shape as data
+
ratio (optional) : T1
-
The ratio of random dropout, with value in [0, 1). If this input was not set, or if it was set to 0, the output would be a simple copy of the input. If it's non-zero, output will be a random dropout of the scaled input, which is typically the case during training. It is an optional value, if not specified it will default to 0.5.
+
training_mode (optional) : T2
-
If set to true then it indicates dropout is being used for training. It is an optional value hence unless specified explicitly, it is false. If it is false, ratio is ignored and the operation mimics inference mode where nothing will be dropped from the input data and if mask is requested as output it will contain all ones.
+
#### Outputs (1 - 2)
output : T
-
The output.
+
mask (optional) : T2
-
The output mask of dropout.
+
#### Type Constraints @@ -588,9 +450,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.BiasGelu** - Bias Gelu. - It's an extension of Gelu. It takes the sum of input A and bias input B as the input of Gelu activation. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -599,16 +458,16 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T
-
The normal input data.
+
B : T
-
The bias input data that is a 1D tensor.
+
#### Outputs
C : T
-
The output.
+
#### Type Constraints @@ -621,8 +480,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.BiasSoftmax** - Y = softmax(scores + bias)) with simple broadcast on bias. Intended to specialize softmax(scores + additive_mask) commonly found in transformer models. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -640,16 +497,16 @@ This version of the operator has been available since version 1 of the 'com.micr
data : T
-
The input data as Tensor.
+
bias : T
-
The bias (or mask) as Tensor.
+
#### Outputs
output : T
-
The output.
+
#### Type Constraints @@ -662,9 +519,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.BiasSplitGelu** - A fusion used in diffusion model that after adding bias, hidden state is sliced into two tensors of same size, then left - tensor multiplies the Gelu activation result of right tensor. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -673,16 +527,16 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
Input tensor. Dimensions are (N, S, D), where N is the batch size, S are image size, and D is hidden dimension
+
bias : T
-
Bias tensor. Dimensions are (D), where D is the same hidden dimension as input tensor
+
#### Outputs
Y : T
-
The output tensor with dimensions (N, S, D/2)
+
#### Type Constraints @@ -695,17 +549,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.BifurcationDetector** - Component for aggressive decoding. Find the bifurcation index of predicted tokens, between source tokens, - starting from previous suffix match index, and predicted tokens. - Concat predicted tokens, starting from bifurcation index, to the back - of current tokens. This forms the output tokens. - Detect suffix match index in source tokens, between source tokens and output tokens. - Detection is based on finding the appearances of last n-gram in output tokens - in source tokens. - A match is considered found if source tokens contain a single matching n-gram. - Return the index of the start of the n-gram in source tokens. - No matching if found if src tokens contain multiple or zero matching n-grams. Return -1. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -723,22 +566,22 @@ This version of the operator has been available since version 1 of the 'com.micr
src_tokens : T
-
Encoder input ids.
+
cur_tokens : T
-
Decoder input ids.
+
prev_suffix_match_idx : T
-
Previous suffix match index
+
pred_tokens (optional) : T
-
Predicted token ids from aggressive decoding
+
#### Outputs
tokens : T
-
Decoder input ids after merging predicted tokens
+
suffix_match_idx : T
-
new suffix match index
+
#### Type Constraints @@ -751,8 +594,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.BitmaskBiasDropout** - output, dropout_bitmask = Dropout(data + bias, ratio) + residual, Intended to specialize the dropout pattern commonly found in transformer models. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -768,24 +609,24 @@ This version of the operator has been available since version 1 of the 'com.micr
data : T
-
The input data as Tensor.
+
bias : T
-
The bias input, a vector with the same shape as last dim of data OR same shape with data
+
residual (optional) : T
-
The residual input, must have the same shape as data
+
ratio (optional) : T1
-
The ratio of random dropout, with value in [0, 1). If this input was not set, or if it was set to 0, the output would be a simple copy of the input. If it's non-zero, output will be a random dropout of the scaled input, which is typically the case during training. It is an optional value, if not specified it will default to 0.5.
+
training_mode (optional) : T2
-
If set to true then it indicates dropout is being used for training. It is an optional value hence unless specified explicitly, it is false. If it is false, ratio is ignored and the operation mimics inference mode where nothing will be dropped from the input data and if mask is requested as output it will contain all ones.
+
#### Outputs (1 - 2)
output : T
-
The output.
+
mask (optional) : T3
-
The output mask of dropout.
+
#### Type Constraints @@ -804,19 +645,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.BitmaskDropout** - BitmaskDropout takes an input floating-point tensor, an optional input ratio (floating-point scalar) and an optional input training_mode (boolean scalar). - It produces two tensor outputs: output (floating-point tensor) and mask (optional `Tensor`). If `training_mode` is true then the output Y will be a random dropout. - Note that this Dropout scales the masked input data by the following equation, so to convert the trained model into inference mode, the user can simply not pass `training_mode` input or set it to false. - ``` - output = scale * data * mask, - ``` - where - ``` - scale = 1. / (1. - ratio). - ``` - - This op functions in much the same was as Dropout-11 and Dropout-13 do, execpt that the mask is output as a bit-packed uint32 tensor, instead of a boolean tensor. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -832,20 +660,20 @@ This version of the operator has been available since version 1 of the 'com.micr
data : T
-
The input data as Tensor.
+
ratio (optional) : T1
-
The ratio of random dropout, with value in [0, 1). If this input was not set, or if it was set to 0, the output would be a simple copy of the input. If it's non-zero, output will be a random dropout of the scaled input, which is typically the case during training. It is an optional value, if not specified it will default to 0.5.
+
training_mode (optional) : T2
-
If set to true then it indicates dropout is being used for training. It is an optional value hence unless specified explicitly, it is false. If it is false, ratio is ignored and the operation mimics inference mode where nothing will be dropped from the input data and if mask is requested as output it will contain all ones.
+
#### Outputs (1 - 2)
output : T
-
The output.
+
mask (optional) : T3
-
The bit-packed output mask.
+
#### Type Constraints @@ -879,16 +707,16 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T
-
2D matrix with shape (M,N)
+
B : T
-
2D matrix with shape (K,N)
+
#### Outputs
C : T
-
A 2D Matrix that represents the distance between each pair of the two collections of inputs.
+
#### Type Constraints @@ -909,16 +737,16 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T
-
input_0
+
B : T
-
input_1
+
#### Outputs
C : T
-
output tensor
+
#### Type Constraints @@ -939,16 +767,16 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T
-
input_0
+
B : T
-
input_1
+
#### Outputs
C : T
-
output tensor
+
#### Type Constraints @@ -1012,13 +840,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.CropAndResize** - Extracts crops from the input image tensor and resizes them using bilinear sampling or nearest neighbor sampling - (possibly with aspect ratio change) to a common output size specified by crop_height and crop_width. - Returns a tensor with crops from the input image at positions defined at the bounding box locations in boxes. - The cropped boxes are all resized (with bilinear or nearest neighbor interpolation) to - a fixed size = [crop_height, crop_width]. The result is a 4-D tensor [num_boxes, crop_height, crop_width, depth]. - The resizing is corner aligned. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1036,20 +857,20 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T1
-
Input data tensor from the previous operator; 4-D feature map of shape (N, C, H, W), where N is the batch size, C is the number of channels, and H and W are the height and the width of the data.
+
rois : T1
-
RoIs (Regions of Interest) to pool over; rois is 2-D input of shape (num_rois, 4) given as [[y1, x1, y2, x2], ...]. The RoIs' coordinates are normalized in the coordinate system of the input image. Each coordinate set has a 1:1 correspondence with the 'batch_indices' input.
+
batch_indices : T2
-
1-D tensor of shape (num_rois,) with each element denoting the index of the corresponding image in the batch.
+
crop_size : T2
-
1-D tensor of 2 elements: [crop_height, crop_width]. All cropped image patches are resized to this size. Both crop_height and crop_width need to be positive.
+
#### Outputs
Y : T1
-
RoI pooled output, 4-D tensor of shape (num_rois, C, crop_height, crop_width). The r-th batch element Y[r-1] is a pooled feature map corresponding to the r-th RoI X[r-1].
+
#### Type Constraints @@ -1064,9 +885,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.DecoderAttention** - This DecoderAttention supports self attention and cross attention, key and value cache, and key_padding_mask. The attention mask is not support at the moment. - Some boolean parameters are passed by runtime input for generic purpose - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1084,40 +902,40 @@ This version of the operator has been available since version 1 of the 'com.micr
query : T
-
3D input tensor with shape (sequence_length, batch_size, hidden_size), hidden_size = num_heads * head_size
+
key : T
-
3D input tensor with shape (total_sequence_length, batch_size, hidden_size)
+
q_weight : T
-
2D input tensor with shape (hidden_size, hidden_size)
+
kv_weight : T
-
2D input tensor with shape (hidden_size, 2 * hidden_size)
+
bias : T
-
1D input tensor with shape (3 * hidden_size)
+
key_padding_mask (optional) : B
-
2D input tensor with shape (batch_size, total_sequence_length)
+
key_cache (optional) : T
-
input tensor with shape (batch_size, num_heads, sequence_length or total_sequence_length, head_size)
+
value_cache (optional) : T
-
input tensor with shape (batch_size, num_heads, sequence_length or total_sequence_length, head_size)
+
static_kv : B
-
If static_kv = true, cross-attention; else self-attention
+
use_past : B
-
If use_past = true, use cache; else no cache
+
has_layer_state : B
-
If has_layer_state = true, layer_state = {} or [a,b]; else layer_state = None
+
has_key_padding_mask : B
-
has_key_padding_mask or not
+
#### Outputs (1 - 3)
output : T
-
3D output tensor with shape (sequence_length, batch_size, hidden_size)
+
new_key_cache (optional) : T
-
output tensor with shape (batch_size, num_heads, new sequence_length, head_size)
+
new_value_cache (optional) : T
-
output tensor with shape (batch_size, num_heads, new sequence_length, head_size)
+
#### Type Constraints @@ -1132,10 +950,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.DecoderMaskedMultiHeadAttention** - Multihead attention that supports input sequence length of 1. - Similar to DecoderMaskedSelfAttention but this op excludes QKV MatMul and Bias. - This op supports both Self and Cross Attention. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1159,40 +973,40 @@ This version of the operator has been available since version 1 of the 'com.micr
query : T
-
Query with shape (batch_size, 1, hidden_size) or packed QKV with shape (batch_size, 1, 2 * hidden_size + v_hidden_size)
+
key (optional) : T
-
Key with shape (batch_size, 1, hidden_size) for self attention or past_key with shape (batch_size, num_heads, kv_sequence_length, head_size) for cross attention
+
value (optional) : T
-
Value with shape (batch_size, 1, v_hidden_size) for self attention or past_value with shape (batch_size, num_heads, kv_sequence_length, head_size) for cross attention
+
mask_index (optional) : M
-
Mask values of shape (batch_size, total_sequence_length) or (batch_size, kv_sequence_length)
+
attention_bias (optional) : T
-
additional add to QxK' with shape (batch_size or 1, num_heads or 1, sequence_length, total_sequence_length)
+
past_key (optional) : T
-
past state for key with shape (batch_size, num_heads, past_sequence_length, head_size) for self attentionWhen past_present_share_buffer is set, its shape is (batch_size, num_heads, max_sequence_length, head_size). The keys buffer is re-ordered in such a way that its virtual sub-tensor of shape (batch_size, num_heads, max_sequence_length, head_size) which may be perceived as being of shape (batch_size, num_heads, max_sequence_length, head_size / x, x) is reordered to become (batch_size, num_heads, head_size / x, max_sequence_length, x) where `x = 16 / sizeof(T)`.
+
past_value (optional) : T
-
past state for value with shape (batch_size, num_heads, past_sequence_length, head_size) for self attentionWhen past_present_share_buffer is set, its shape is (batch_size, num_heads, max_sequence_length, head_size).
+
past_sequence_length (optional) : M
-
When past_present_share_buffer is used, it is required to specify past_sequence_length (could be 0).Cross Attention doesn't need this input.
+
beam_width (optional) : M
-
The beam width that is being used while decoding. If not provided, the beam width will be assumed to be 1.
+
cache_indirection (optional) : M
-
A buffer of shape [batch_size, beam_width, max_output_length] where an `[i, j, k]` entry specifies which beam the `k`-th token came from for the `j`-th beam for batch `i` in the current iteration
+
bias (optional) : T
-
Bias tensor with shape (hidden_size + hidden_size + v_hidden_size) from input projection
+
#### Outputs (1 - 4)
output : T
-
3D output tensor with shape (batch_size, sequence_length, v_hidden_size)
+
present_key (optional) : T
-
present state for key with shape (batch_size, num_heads, total_sequence_length, head_size). If past_present_share_buffer is set, its shape is (batch_size, num_heads, max_sequence_length, head_size), while effective_seq_length = (past_sequence_length + kv_sequence_length).
+
present_value (optional) : T
-
present state for value with shape (batch_size, num_heads, total_sequence_length, head_size). If past_present_share_buffer is set, its shape is (batch_size, num_heads, max_sequence_length, head_size), while effective_seq_length = (past_sequence_length + kv_sequence_length).
+
qk (optional) : V
-
normalized Q * K, of shape (batch_size, num_heads, 1, total_sequence_length).
+
#### Type Constraints @@ -1209,21 +1023,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.DecoderMaskedSelfAttention** - Self attention that supports input sequence length of 1. - - The weights for input projection of Q, K and V are merged. The data is stacked on the second dimension. Its shape - is (input_hidden_size, hidden_size + hidden_size + v_hidden_size). Here hidden_size is the hidden dimension of Q and K, - and v_hidden_size is that of V. - - The mask_index is optional. If it is provided, only raw attention mask with shape (batch_size, total_sequence_length) is supported currently. - - Both past and present state need to be provided. - - The qkv_hidden_sizes is required only when K and V have different hidden sizes. - - The total_sequence_length is past_sequence_length + kv_sequence_length. Here kv_sequence_length is the length of K or V. - Currently, only self attention is supported which means that kv_sequence_length equals to sequence_length (sequence length of Q). - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1247,32 +1046,32 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T
-
Input tensor with shape (batch_size, 1, input_hidden_size)
+
weights : T
-
Merged Q/K/V weights with shape (input_hidden_size, hidden_size + hidden_size + v_hidden_size)
+
bias : T
-
Bias tensor with shape (hidden_size + hidden_size + v_hidden_size) for input projection
+
mask_index (optional) : M
-
Mask values of shape (batch_size, total_sequence_length)
+
past : T
-
past state for key and value with shape (2, batch_size, num_heads, past_sequence_length, head_size)When past_present_share_buffer is set, its shape is (2, batch_size, num_heads, max_sequence_length, head_size). The first `batch_size * num_heads * max_sequence_length * head_size` elements correspond to keys and the next `batch_size * num_heads * max_sequence_length * head_size` elements correspond to values. The keys buffer is re-ordered in such a way that its virtual sub-tensor of shape (batch_size, num_heads, max_sequence_length, head_size) which may be perceived as being of shape (batch_size, num_heads, max_sequence_length, head_size / x, x) is reordered to become (batch_size, num_heads, head_size / x, max_sequence_length, x) where `x = 16 / sizeof(T)`.
+
attention_bias (optional) : T
-
additional add to QxK' with shape (batch_size or 1, num_heads or 1, sequence_length, total_sequence_length)
+
past_sequence_length : M
-
When past_present_share_buffer is used, it is required to specify past_sequence_length (could be 0).
+
beam_width (optional) : M
-
The beam width that is being used while decoding. If not provided, the beam width will be assumed to be 1.
+
cache_indirection (optional) : M
-
A buffer of shape [batch_size, beam_width, max_output_length] where an `[i, j, k]` entry specifies which beam the `k`-th token came from for the `j`-th beam for batch `i` in the current iteration
+
#### Outputs
output : T
-
3D output tensor with shape (batch_size, sequence_length, v_hidden_size)
+
present : T
-
past state for key and value with shape (2, batch_size, num_heads, total_sequence_length, head_size). If past_present_share_buffer is set, its shape is (2, batch_size, num_heads, max_sequence_length, head_size), while effective_seq_length = (past_sequence_length + kv_sequence_length).
+
#### Type Constraints @@ -1287,10 +1086,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.DequantizeBFP** - The BFP dequantization operator. - It consumes the raw BFP data and some metadata such as the shape and strides of the original tensor and computes the dequantized tensor. - More documentation on the BFP format can be found in this paper: https://www.microsoft.com/en-us/research/publication/pushing-the-limits-of-narrow-precision-inferencing-at-cloud-scale-with-microsoft-floating-point/ - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1310,18 +1105,18 @@ This version of the operator has been available since version 1 of the 'com.micr
x : T1
-
1-D, contiguous, raw, BFP data to be de-quantized.
+
shape : T2
-
shape of the original tensor.
+
strides : T2
-
strides of the original tensor.
+
#### Outputs
y : T3
-
de-quantized tensor.
+
#### Type Constraints @@ -1338,10 +1133,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.DequantizeLinear** - The linear dequantization operator. It consumes a quantized data, a scale, a zero point and computes the full precision data. - The dequantization formula is y = (x - x_zero_point) * x_scale. - Scale and zero point must have same shape. They must be either scalar (per tensor) or 1-D tensor (per 'axis'). - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1357,18 +1148,18 @@ This version of the operator has been available since version 1 of the 'com.micr
x : T1
-
N-D quantized Input tensor to be de-quantized.
+
x_scale : T2
-
Scale for input 'x'. It can be a scalar, which means a per-tensor/layer dequantization, or a 1-D tensor for per-axis dequantization.
+
x_zero_point (optional) : T1
-
Zero point for input 'x'. Shape must match x_scale. It's optional. Zero point is 0 when it's not specified.
+
#### Outputs
y : T2
-
N-D full precision output tensor. It has same shape as input 'x'.
+
#### Type Constraints @@ -1383,8 +1174,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.DequantizeWithOrder** - Dequantize input matrix to specific layout used in cublaslt. attr to specify output type, float16 or float32 - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1404,16 +1193,16 @@ This version of the operator has been available since version 1 of the 'com.micr
input : Q
-
TODO: input tensor of (ROWS, COLS). if less than 2d, will broadcast to (1, X). If 3d, it is treated as (B, ROWS, COS)
+
scale_input : S
-
scale of the input
+
#### Outputs
output : F
-
output tensor
+
#### Type Constraints @@ -1457,40 +1246,40 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
The input sequences packed (and potentially padded) into one 3-D tensor with the shape of `[seq_length, batch_size, input_size]`.
+
W : T2
-
The weight tensor for the gates. Concatenation of `W[iofc]` and `WB[iofc]` (if bidirectional) along dimension 0. The tensor has shape `[num_directions, input_size, 4*hidden_size]`.
+
R : T2
-
The recurrence weight tensor. Concatenation of `R[iofc]` and `RB[iofc]` (if bidirectional) along dimension 0. This tensor has shape `[num_directions, hidden_size, 4*hidden_size]`.
+
B (optional) : T
-
The bias tensor for input gate. Concatenation of `[Wb[iofc], Rb[iofc]]`, and `[WBb[iofc], RBb[iofc]]` (if bidirectional) along dimension 0. This tensor has shape `[num_directions, 8*hidden_size]`. Optional: If not specified - assumed to be 0.
+
sequence_lens (optional) : T1
-
Optional tensor specifying lengths of the sequences in a batch. If not specified - assumed all sequences in the batch to have length `seq_length`. It has shape `[batch_size]`.
+
initial_h (optional) : T
-
Optional initial value of the hidden. If not specified - assumed to be 0. It has shape `[num_directions, batch_size, hidden_size]`.
+
initial_c (optional) : T
-
Optional initial value of the cell. If not specified - assumed to be 0. It has shape `[num_directions, batch_size, hidden_size]`.
+
P (optional) : T
-
The weight tensor for peepholes. Concatenation of `P[iof]` and `PB[iof]` (if bidirectional) along dimension 0. It has shape `[num_directions, 3*hidde_size]`. Optional: If not specified - assumed to be 0.
+
W_scale : T
-
W's scale. Its size is [num_directions] for per-tensor/layer quantization, or [num_directions, 4*hidden_size] for per-channel quantization on the axis input_size.
+
W_zero_point : T2
-
W's zero point. Its size is [num_directions] for per-tensor/layer quantization, or [num_directions, 4*hidden_size] for per-channel quantization on the axis input_size.
+
R_scale : T
-
R's scale. Its size is [num_directions] for per-tensor/layer quantization, or [num_directions, 4*hidden_size] for per-channel quantization on the axis input_size.
+
R_zero_point : T2
-
R's zero point. Its size is [num_directions] for per-tensor/layer quantization, or [num_directions, 4*hidden_size] for per-channel quantization on the axis input_size.
+
#### Outputs (0 - 3)
Y (optional) : T
-
A tensor that concats all the intermediate output values of the hidden. It has shape `[seq_length, num_directions, batch_size, hidden_size]`.
+
Y_h (optional) : T
-
The last output value of the hidden. It has shape `[num_directions, batch_size, hidden_size]`.
+
Y_c (optional) : T
-
The last output value of the cell. It has shape `[num_directions, batch_size, hidden_size]`.
+
#### Type Constraints @@ -1515,22 +1304,22 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T1
-
N-dimensional matrix A
+
B : T2
-
N-dimensional matrix B
+
b_scale : T1
-
Scale of quantized input 'B'. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'B'.
+
b_zero_point (optional) : T2
-
Zero point tensor for input 'B'. It's optional and default value is 0. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'B'.
+
bias (optional) : T1
-
1D input tensor, whose dimension is same as B's last dimension
+
#### Outputs
Y : T1
-
Matrix multiply results from A * B
+
#### Type Constraints @@ -1545,8 +1334,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.DynamicTimeWarping** - Input is cost matrix where each value in input[r][c] is the cost for pass the point (r, c). From current point(r, c), points (r+1, c), (r+1, c+1) or (r, c+1) could be arrived in next move. Given such cost matrix, return dynamic time warping of shape [2, x], where the path made by all points (output[0][t], output[1][t])have the lowest cost among all paths from (0, 0) to (M-1, N-1). - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1555,14 +1342,14 @@ This version of the operator has been available since version 1 of the 'com.micr
input : F
-
Input cost tensor, it must be 2D tensor of shape M x N, or 1 x M x N
+
#### Outputs
output : I
-
Output tensor. shape is [2, x], where max(M, N) <= x < M + N
+
#### Type Constraints @@ -1577,8 +1364,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.EPContext** - Onnx node container for EP context. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1612,32 +1397,26 @@ This version of the operator has been available since version 1 of the 'com.micr
inputs (variadic, heterogeneous) : T
-
List of tensors for inputs
+
#### Outputs (1 - ∞)
outputs (variadic, heterogeneous) : T
-
One or more outputs, list of tensors for outputs
+
#### Type Constraints
-
T : tensor(int8), tensor(int16), tensor(int32), tensor(int64), tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
+
T : tensor(int8), tensor(int16), tensor(int32), tensor(int64), tensor(bool), tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
Constrain input and output types.
### **com.microsoft.EmbedLayerNormalization** - EmbedLayerNormalization is the fusion of embedding layer in BERT model, with optional mask processing. - The embedding layer takes input_ids (word IDs) and segment_ids (sentence IDs) to look up word_embedding, position_embedding, - and segment_emedding; the embeddings are added then applied layer normalization using gamma and beta tensors. - The last input mask is optional. If mask is provided, mask index (that is position of first 0 in mask, or number of words) - will be calculated. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1655,34 +1434,34 @@ This version of the operator has been available since version 1 of the 'com.micr
input_ids : T1
-
2D words IDs with shape (batch_size, sequence_length)
+
segment_ids (optional) : T1
-
2D segment IDs with shape (batch_size, sequence_length)
+
word_embedding : T
-
2D with shape (,hidden_size)
+
position_embedding : T
-
2D with shape (, hidden_size)
+
segment_embedding (optional) : T
-
2D with shape (, hidden_size)
+
gamma : T
-
1D gamma tensor for layer normalization with shape (hidden_size)
+
beta : T
-
1D beta tensor for layer normalization with shape (hidden_size)
+
mask (optional) : T1
-
2D attention mask with shape (batch_size, sequence_length)
+
position_ids (optional) : T1
-
2D position ids with shape (batch_size, sequence_length) or (1, sequence_length)
+
#### Outputs (1 - 3)
output : T
-
3D output tensor with shape (batch_size, sequence_length, hidden_size)
+
mask_index (optional) : T1
-
1D mask_index tensor with shape (batch_size)
+
embedding_sum (optional) : T
-
sum of word_embedding and position_embedding without layer normalization
+
#### Type Constraints @@ -1697,8 +1476,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.ExpandDims** - ExpandDims echo operator. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1707,16 +1484,16 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
input
+
axis : tensor(int32)
-
Specified axis to insert a dimension
+
#### Outputs
Y : T
-
output
+
#### Type Constraints @@ -1729,8 +1506,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.FastGelu** - GELU (Gaussian Error Linear Unit) approximation: Y=0.5*X*(1+tanh(0.797885*X+0.035677*X*X*X)) with an optional input of bias that will be added to X before GELU. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1739,16 +1514,16 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
input tensor
+
bias (optional) : T
-
bias tensor
+
#### Outputs
Y : T
-
output tensor
+
#### Type Constraints @@ -1761,9 +1536,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.FusedConv** - The fused convolution operator schema is the same as Conv besides it includes an attribute - activation. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1819,9 +1591,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.FusedGemm** - The FusedGemm operator schema is the same as Gemm besides it includes attributes - activation and leaky_relu_alpha. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1851,18 +1620,18 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T
-
Input tensor A. The shape of A should be (M, K) if transA is 0, or (K, M) if transA is non-zero.
+
B : T
-
Input tensor B. The shape of B should be (K, N) if transB is 0, or (N, K) if transB is non-zero.
+
C (optional) : T
-
Input tensor C. The shape of C should be unidirectional broadcastable to (M, N).
+
#### Outputs
Y : T
-
Output tensor of shape (M, N).
+
#### Type Constraints @@ -1875,8 +1644,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.FusedMatMul** - Matrix product that behaves like numpy.matmul: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.matmul.html - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1900,16 +1667,16 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T
-
N-dimensional matrix A
+
B : T
-
N-dimensional matrix B
+
#### Outputs
Y : T
-
Matrix multiply results
+
#### Type Constraints @@ -1922,8 +1689,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.FusedMatMulActivation** - Executes the same operation as FusedMatMul, but also has an activation function fused to its output. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -1957,16 +1722,16 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T
-
N-dimensional matrix A
+
B : T
-
N-dimensional matrix B
+
#### Outputs
Y : T
-
Matrix multiply results
+
#### Type Constraints @@ -1979,13 +1744,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.GatedRelativePositionBias** - query_layer = (query_layer + query_bias).reshape(batch_size, seq_len, num_heads, head_size).transpose(1, 2) - gate_u, gate_r = torch.sigmoid( - self.gate_ur_linear(query_layer).view(batch_size, num_head, seq_len, 2, D/2).sum(-1, keepdim=False) - ).chunk(2, dim=-1) - gate_u_1 = gate_u * (gate_r * self.eco_a - 1.0) + 2.0 - rel_pos_bias = gate_u_1 * rel_pos - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2001,26 +1759,26 @@ This version of the operator has been available since version 1 of the 'com.micr
query_layer : T
-
tensor with shape (batch_size, seq_len, num_heads x head_size) or (token_count, num_heads x head_size)
+
query_bias : T
-
1-d tensor with shape (num_heads x head_size)
+
rel_pos : T
-
tensor with shape (1, num_head, seq_len, seq_len)
+
weight : T
-
gemm weight for the gated_ur_linear, shape (head_size, D), D is divisible by 2
+
bias : T
-
bias for the gated_ur_linear, shape (D)
+
eco_a : T
-
tensor of shape (1, num_heads, 1, 1)
+
token_offset (optional) : M
-
offset of each token with shape (batch_size, seq_len)
+
#### Outputs
output : T
-
output tensor with shape (batch_size, num_heads, seq_len, seq_len)
+
#### Type Constraints @@ -2035,15 +1793,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.GatherBlockQuantized** - GatherBlockQuantized is a Gather with data quantized. It is similar to Gather (https://github.com/onnx/onnx/blob/main/docs/Operators.md#gather) with differences: - 1. Input `data` is a constant. It is quantized block-wise along attribute `quantize_axis` with block size specified by attribute `block_size`. - `block_size must` be a power of 2 and not smaller than 16, like 16, 32, 64, 128, .. - 2. Input `data`'s scale and zero point are specified by input `scales` and `zero_points`. `scales` and `zero_points` are also constants. - If `zero_points` is not provided, 0 is the zero point. - 3. During the op execution, `data` and `indices` are first used to generate the quantized output. Then, `scales` and `zero_points` are used - to dequantize the output. - 4. The `output` and `scales` have the same type. The `data` and `zero_points` have the same type. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2063,20 +1812,20 @@ This version of the operator has been available since version 1 of the 'com.micr
data : T1
-
Tensor of rank r >= 1. Block-wise quantized.
+
indices : Tind
-
Tensor of int32/int64 indices, of any rank q. All index values are expected to be within bounds [-s, s-1] along axis of size s. It is an error if any of the index values are out of bounds.
+
scales : T2
-
quantization scale
+
zero_points (optional) : T1
-
quantization zero points
+
#### Outputs
output : T2
-
Dequantized output tensor of rank q + (r - 1).
+
#### Type Constraints @@ -2093,25 +1842,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.GatherND** - Given `data` tensor of rank r >= 1, and `indices` tensor of rank q >= 1, gather - slices of `data` into an output tensor of rank q - 1 + r - indices[-1]. - Example 1: - data = [[0,1],[2,3]] - indices = [[0,0],[1,1]] - output = [0,3] - Example 2: - data = [[0,1],[2,3]] - indices = [[1],[0]] - output = [[2,3],[0,1]] - Example 3: - data = [[[0,1],[2,3]],[[4,5],[6,7]]] - indices = [[0,1],[1,0]] - output = [[2,3],[4,5]] - Example 4: - data = [[[0,1],[2,3]],[[4,5],[6,7]]] - indices = [[[0,1]],[[1,0]]] - output = [[[2,3]],[[4,5]]] - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2120,16 +1850,16 @@ This version of the operator has been available since version 1 of the 'com.micr
data : T
-
Tensor of rank r >= 1.
+
indices : Tind
-
Tensor of rank q >= 1.
+
#### Outputs
output : T
-
Tensor of rank q-1+r-indices[-1].
+
#### Type Constraints @@ -2144,12 +1874,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.Gelu** - Gaussian Error Linear Unit. - A high-performing neural network activation function.The GELU nonlinearity is - the expected transformation of a stochastic regularizer which randomly applies - the identity or zero map to a neuron's input. The GELU nonlinearity weights - inputs by their magnitude, rather than gates inputs by their sign as in ReLUs. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2158,14 +1882,14 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
The input data as Tensor.
+
#### Outputs
Y : T
-
The output.
+
#### Type Constraints @@ -2178,8 +1902,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.GemmFastGelu** - It's a fusion of MatMul and FastGelu. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2188,18 +1910,18 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
input tensor
+
W : T
-
input tensor
+
bias (optional) : T
-
bias tensor
+
#### Outputs
Y : T
-
output tensor
+
#### Type Constraints @@ -2212,8 +1934,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.GemmFloat8** - Generic Gemm for float and float 8. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2239,24 +1959,24 @@ This version of the operator has been available since version 1 of the 'com.micr
A : TA
-
Input tensor A. The shape of A should be (M, K) if transA is 0, or (K, M) if transA is non-zero.
+
B : TB
-
Input tensor B. The shape of B should be (K, N) if transB is 0, or (N, K) if transB is non-zero.
+
C (optional) : TC
-
Input tensor C.
+
scaleA (optional) : TS
-
Scale of tensor A if A is float 8 tensor
+
scaleB (optional) : TS
-
Scale of tensor B if B is float 8 tensor
+
scaleY (optional) : TS
-
Scale of the output tensor if A or B is float 8.
+
#### Outputs
Y : TR
-
Output tensor of shape (M, N).
+
#### Type Constraints @@ -2277,29 +1997,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.GemmaRotaryEmbedding** - GemmaRotaryEmbedding is the implementation of below part of rotary positional embeddings (RoPE). It implements below from modeling_gemma.py. - - Here's onnxscript that was tested - - from onnxscript import FLOAT, FLOAT16, script - from onnxscript import opset18 as op - - @script() - def gemma_rotary_embedding(emb: FLOAT["bs", "seq_len", "dim"], q: FLOAT16["bs", "num_heads", "seq_len", "dim"], q_rot: FLOAT16["bs", "num_heads", "seq_len", "dim"], k: FLOAT16["bs", "num_heads", "seq_len", "dim"], k_rot: FLOAT16["bs", "num_heads", "seq_len", "dim"]): - sin_val = op.Sin(emb) - casted_sin = op.Cast(sin_val, to=10) # for fp16 mix-precision training. Other types are not supported. - cos_val = op.Cos(emb) - casted_cos = op.Cast(cos_val, to=10) - unsqueezed_sin = op.Unsqueeze(casted_sin, [1]) - unsqueezed_cos = op.Unsqueeze(casted_cos, [1]) - q_embed = (q * casted_cos) + (q_rot * casted_sin) - k_embed = (k * casted_cos) + (k_rot * casted_sin) - return q_embed, k_embed - - onnx_model = gemma_rotary_embedding.to_model_proto() - - - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2308,24 +2005,24 @@ This version of the operator has been available since version 1 of the 'com.micr
emb : U
-
embeddding - 3D tensor with shape (batch_size, seq_len, dim)
+
q : T
-
q state - 4D tensor with shape (batch_size, num_heads, seq_len, dim)
+
q_rot : T
-
half rotated q state - 4D tensor with shape (batch_size, num_heads, seq_len, dim)
+
k : T
-
k state - 4D tensor with shape (batch_size, num_heads, seq_len, dim)
+
k_rot : T
-
k state - 4D tensor with shape (batch_size, num_heads, seq_len, dim)
+
#### Outputs
output1 : T
-
4D tensor with shape (batch_size, num_heads, seq_len, dim)
+
output2 : T
-
4D tensor with shape (batch_size, num_heads, seq_len, dim)
+
#### Type Constraints @@ -2340,8 +2037,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.GreedySearch** - Greedy Search for text generation. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2373,26 +2068,26 @@ This version of the operator has been available since version 1 of the 'com.micr
input_ids : I
-
The sequence used as a prompt for the generation. Shape is (batch_size, sequence_length)
+
max_length : I
-
The maximum length of the sequence to be generated. Shape is (1)
+
min_length (optional) : I
-
The minimum length below which the score of eos_token_id is set to -Inf. Shape is (1)
+
repetition_penalty (optional) : T
-
The parameter for repetition penalty. Default value 1.0 means no penalty. Accepts value > 0.0. Shape is (1)
+
vocab_mask (optional) : I
-
Mask of vocabulary. Words that masked with 0 are not allowed to be generated, and 1 is allowed. Shape is (vocab_size)
+
prefix_vocab_mask (optional) : I
-
Mask of vocabulary for first step. Words that masked with 0 are not allowed to be generated, and 1 is allowed. Shape is (batch_size, vocab_size)
+
attention_mask (optional) : I
-
Custom attention mask. Shape is (batch_size, sequence_length)
+
#### Outputs
sequences : I
-
Word IDs of generated sequences. Shape is (batch_size, max_sequence_length)
+
#### Type Constraints @@ -2407,15 +2102,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.GridSample** - Given an `input` and a flow-field `grid`, computes the `output` using `input` values and pixel locations from `grid`. - Currently, only spatial (4-D) inputs are supported. For `input` with shape (N, C, H, W) and `grid` with shape (N, H_out, W_out, 2), - the `output` will have shape (N, C, H_out, W_out). - For each output location `output[n, :, h, w]`, the size-2 vector `grid[n, h, w]` specifies `input` pixel locations `x` and `y`, - which are used to interpolate the output value `output[n, :, h, w]`. - The GridSample operator is often used in doing grid generator and sampler in the [Spatial Transformer Networks](https://arxiv.org/abs/1506.02025). - See also in [torch.nn.functional.grid_sample](https://pytorch.org/docs/master/generated/torch.nn.functional.grid_sample.html#torch-nn-functional-grid-sample). - - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2435,16 +2121,16 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T1
-
4-D tensor of shape (N, C, H, W), where N is the batch size, C is the numbers of channels, H and W are the height and width of the input data.
+
Grid : T1
-
Input offset, 4-D tensor of shape (N, H_out, W_out, 2), where H_out and W_out are the height and width of grid and output, Grid specifies the sampling pixel locations normalized by the input spatial dimensions. Therefore, it should have most values in the range of [-1, 1]. If grid has values outside the range of [-1, 1], the corresponding outputs will be handled as defined by padding_mode.
+
#### Outputs
Y : T2
-
4-D tensor of shape (N, C, H_out, W_out).
+
#### Type Constraints @@ -2459,16 +2145,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.GroupNorm** - Applies Group Normalization over a mini-batch of inputs as described in the paper Group Normalization (https://arxiv.org/abs/1803.08494). - - This operator transforms input according to - y = gamma * (x - mean) / sqrt(variance + epsilon) + beta - - The input channels are separated into num_groups groups, each containing num_channels / num_groups channels. num_channels must be divisible by num_groups. The mean and standard-deviation are calculated separately over the each group. - The weight and bias are per-channel affine transform parameter vectors of size num_channels. - - The activation attribute can be used to enable activation after group normalization. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2490,18 +2166,18 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
Input data tensor. Dimensions are (N x H x W x C) when channels_last is 1 or (N x C x H x W) otherwise, where N is the batch size, C is the number of channels, and H and W are the height and width of the data
+
gamma : M
-
1D gamma tensor for normalization with shape (C), where C is number of channels
+
beta : M
-
1D beta tensor for normalization with shape (C), where C is number of channels
+
#### Outputs
Y : T
-
The output tensor of the same shape as X
+
#### Type Constraints @@ -2516,16 +2192,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.GroupQueryAttention** - Group Query Self/Cross Attention. - - *Highly recommend using k-v cache share buffer for both CPU and CUDA. Enabled through IOBinding past and present kv. - Supports different number of heads for q and kv for CPU and CUDA. - Only supports causal and local attention. - Supports rotary position embedding for CPU and CUDA. - Supports packed input for CPU and CUDA. - Supports continuous decoding for batch_size == 1 for CPU and CUDA. - - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2555,34 +2221,34 @@ This version of the operator has been available since version 1 of the 'com.micr
query : T
-
Query with shape (batch_size, sequence_length, hidden_size), or packed QKV with shape(batch_size, sequence_length, d) where d is (num_heads * head_size + 2 * kv_num_heads * head_size).
+
key (optional) : T
-
Key with shape (batch_size, kv_sequence_length, kv_hidden_size)
+
value (optional) : T
-
Value with shape (batch_size, kv_sequence_length, kv_hidden_size)
+
past_key (optional) : T
-
past state key with support for format BNSH. When past_key uses same tensor as present_key(k-v cache), it is of length max_sequence_length... otherwise of length past_sequence_length.
+
past_value (optional) : T
-
past state value with support for format BNSH. When past_value uses same tensor as present_value(k-v cache), it is of length max_sequence_length... otherwise of length past_sequence_length.
+
seqlens_k : M
-
1D Tensor of shape (batch_size). Equivalent to (total_sequence_lengths - 1).
+
total_sequence_length : M
-
Scalar tensor equivalent to the maximum total sequence length (past + new) of the batch. Used for checking inputs and determining prompt vs token generation case.
+
cos_cache (optional) : T
-
2D tensor with shape (max_sequence_length, head_size / 2).
+
sin_cache (optional) : T
-
2D tensor with shape (max_sequence_length, head_size / 2).
+
#### Outputs
output : T
-
3D output tensor with shape (batch_size, sequence_length, hidden_size)
+
present_key : T
-
present state key with support for format BNSH. When past_key uses same tensor as present_key(k-v buffer), it is of length max_sequence_length... otherwise of length past_sequence_length +kv_sequence_length.
+
present_value : T
-
present state value with support for format BNSH. When past_value uses same tensor as present_value(k-v buffer), it is of length max_sequence_length... otherwise of length past_sequence_length +kv_sequence_length.
+
#### Type Constraints @@ -2605,14 +2271,14 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
Input tensor. Every matrix in the batch must be invertible.
+
#### Outputs
Y : T
-
Output tensor of the same type and shape as the input tensor.
+
#### Type Constraints @@ -2625,8 +2291,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.Irfft** - This function computes the inverse of the one-dimensional n-point RFFT computed in 'com.microsoft.rfft'. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2646,14 +2310,14 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
input tensor with size (n//2 + 1) in the signal dim and 2 in the last dimension for the real and complex parts
+
#### Outputs
Y : T
-
output tensor with size n in the signal dim
+
#### Type Constraints @@ -2666,15 +2330,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.LongformerAttention** - Longformer Self Attention with a local context and a global context. Tokens attend locally: Each token - attends to its W previous tokens and W succeeding tokens with W being the window length. A selected few tokens - attend globally to all other tokens. - - The attention mask is of shape (batch_size, sequence_length), where sequence_length is a multiple of 2W after padding. - Mask value < 0 (like -10000.0) means the token is masked, 0 otherwise. - - Global attention flags have value 1 for the tokens attend globally and 0 otherwise. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2692,26 +2347,26 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T
-
3D input tensor with shape (batch_size, sequence_length, hidden_size), hidden_size = num_heads * head_size
+
weight : T
-
2D input tensor with shape (hidden_size, 3 * hidden_size)
+
bias : T
-
1D input tensor with shape (3 * hidden_size)
+
mask : T
-
Attention mask with shape (batch_size, sequence_length)
+
global_weight : T
-
2D input tensor with shape (hidden_size, 3 * hidden_size)
+
global_bias : T
-
1D input tensor with shape (3 * hidden_size)
+
global : G
-
Global attention flags with shape (batch_size, sequence_length)
+
#### Outputs
output : T
-
3D output tensor with shape (batch_size, sequence_length, hidden_size)
+
#### Type Constraints @@ -2726,38 +2381,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.MatMulBnb4** - MatMulBnb4 is a MatMul with weight quantized with 4 bits using either FP4 or NF4 data type (https://arxiv.org/pdf/2305.14314.pdf). It does Matrix Multiplication like MatMul (https://github.com/onnx/onnx/blob/main/docs/Operators.md#matmul) with differences: - 1. Input B is a 2D constant Matrix. Its input feature count and output feature count are specified by attribute 'K' and 'N'. - 2. Input B is quantized with 4 bits with quantization data type specified by attribute 'quant_type'. It is transposed, flattened and quantized blockwisely with block size specified by attribute 'block_size'. - And block_size is not an arbitrary number and must be a power of 2 and not smaller than 16, like 16, 32, 64, 128,.. - 3. Input B's quantization constants or scales are specified by input 'absmax'. - - Input B is stored as uint8_t with shape: [(N * K + 1) / 2]. - Input absmax is stored in same type as original type of B(float32, float16) with shape like: [(N * K + block_size - 1) / block_size]. - - - 1. (Default value) transB=True (Majorly used for forward pass) - Shape of A: [D0, D1, ..., Dn, K] - Shape of Dequanted B: [N, K], this is aligned with how PyTorch defined the linear weight, .e.g [out_features, in_features]. - - The computation math: - dequant_B = dequant(B, absmax, quant_type, block_size) - transposed_dequant_B = dequant_B^T - output = A @ transposed_dequant_B - - Shape of output: [D0, D1, ..., Dn, N] - - 2. transB=False (Majorly used for backward pass) - Shape of A: [D0, D1, ..., Dn, N] - Shape of Dequanted B: [N, K], this is aligned with how PyTorch defined the linear weight, .e.g [out_features, in_features]. - - The computation math: - dequant_B = dequant(B, absmax, quant_type, block_size) - output = A @ dequant_B - - Shape of output: [D0, D1, ..., Dn, K] - - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2783,18 +2406,18 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T1
-
The input tensor, not quantized
+
B : T2
-
1-dimensional quantized data for weight
+
absmax : T1
-
quantization constants
+
#### Outputs
Y : T1
-
tensor. The output tensor has the same rank as the input.
+
#### Type Constraints @@ -2809,14 +2432,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.MatMulFpQ4** - Matrix product with right hand matrix being pre-packed and quantized int4 data blob. - During quantization, the matrix is divided into blocks, where each block is a - continguous subset inside each column. Each block is quantized into a - sequence of 4b integers with a scaling factor and an optional offset. - Currently 3 quantization types are supported: - (0): block size 32, no offset, (1): block size 32, with offset, (2): block size 64, - no offset - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2832,18 +2447,18 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T1
-
N-dimensional matrix A
+
B : T2
-
1-dimensional data blob
+
B_shape : T3
-
Shape information of B
+
#### Outputs
Y : T1
-
Matrix multiply results from A * B
+
#### Type Constraints @@ -2860,9 +2475,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.MatMulInteger16** - Matrix product that behaves like numpy.matmul: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.matmul.html. - The production MUST never overflow. The accumulation may overflow if and only if in 32 bits. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2871,16 +2483,16 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T1
-
N-dimensional matrix A
+
B : T2
-
N-dimensional matrix B
+
#### Outputs
Y : T3
-
Matrix multiply results from A * B
+
#### Type Constraints @@ -2905,26 +2517,26 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T1
-
N-dimensional matrix A
+
B : T2
-
N-dimensional matrix B
+
a_scale : T3
-
Scale of quantized input 'A'. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'A'.
+
b_scale : T3
-
Scale of quantized input 'B'. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'B'.
+
a_zero_point (optional) : T1
-
Zero point tensor for input 'A'. It's optional and default value is 0. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'A'.
+
b_zero_point (optional) : T2
-
Zero point tensor for input 'B'. It's optional and default value is 0. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'B'.
+
bias (optional) : T3
-
1D input tensor, whose dimension is same as B's last dimension
+
#### Outputs
Y : T3
-
Matrix multiply results from A * B
+
#### Type Constraints @@ -2941,30 +2553,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.MatMulNBits** - MatMulNBits is a MatMul with weight quantized with N bits(e.g., 2, 3, 4, 5, 6, 7).It does Matrix Multiplication like MatMul (https://github.com/onnx/onnx/blob/main/docs/Operators.md#matmul) with differences: - 1. Input B is a 2D constant Matrix. Its input feature count and output feature count are specified by attribute 'K' and 'N'. - 2. Input B is quantized with x bits which is specified by attribute 'bits'. It is quantized blockwisely along dimension 0 (e.g. column) with block size specified by attribute block_size. - And block_size is not an arbitrary number and must be a power of 2 and not smaller than 16, like 16, 32, 64, 128,.. - 3. Input B's scale and zero point are specified by input scales and zero_points. - - Input B is stored as uint8_t with shape: [N][n_blocks_per_col][blob_size] in which: - - n_blocks_per_col = (K + block_size - 1) / block_size - - blob_size = CeilDiv(block_size * bits, bitsof(uint8_t)<8>) - For all bits from 2-8, a row of data is stored squeezely and represented by uint8_t. - - for 2,4,8 bits, 4x2bit,2x4bit,1x8bit are stored in one uint8_t. - 4bit example: - |.|.|.|.| .|.|.|.| =uint8_t (2x4bit) - - for 3,5,6,7 bits, 32x3bit,32x5bit,16x6bit,32x7bit are stored in 12xuint8_t,20xuint8_t,12xuint8_t,28xuint8_t separately. no bits are wasted. - 3bit example: - |.|.|. |.|.|. |.|.|. = 9bit, which across 2 uint8_t, the highest bit for the second uint8_t is used. - The last uint_8 may have some bits unused. - - - Input scales is stored in same type as original type of B(float32, float16) with shape like: [N * n_blocks_per_col] - Input zero_points is stored as uint8_t or same as type(A). It has the same packing method as input B. - - [N * CeilDiv(n_blocks_per_col * bits, 8)] - If zero_points has same type as A, it's not packed and has the same shape as Scales. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -2988,24 +2576,24 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T1
-
The input tensor, not quantized
+
B : T2
-
1 or 2 dimensional data blob
+
scales : T1
-
quantization scale
+
zero_points (optional) : T3
-
quantization zero points
+
g_idx (optional) : T4
-
group_idx
+
bias (optional) : T1
-
Bias to add to result. It should have shape [N].
+
#### Outputs
Y : T1
-
tensor. The output tensor has the same rank as the input.
+
#### Type Constraints @@ -3024,8 +2612,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.MaxpoolWithMask** - For internal use. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3051,7 +2637,7 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
M : tensor(int32)
-
mask
+
#### Outputs @@ -3071,11 +2657,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.MoE** - Mixture of experts. Examples: Switch transformer(https://arxiv.org/pdf/2101.03961.pdf) use top 1, - GLaM(https://arxiv.org/abs/2112.06905) activates top 2 FFN, Vision MOE(https://arxiv.org/pdf/2106.05974.pdf) - usually uses top 32 experts and Mixtral(https://huggingface.co/blog/mixtral). - - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3097,28 +2678,28 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T
-
2D input tensor with shape (num_rows, hidden_size) or 3D input tensor with shape (batch_size, sequence_length, hidden_size)
+
router_probs : T
-
2D input tensor with shape (num_rows, num_experts)
+
fc1_experts_weights : T
-
3D input tensor with shape (num_experts, hidden_size, inter_size)
+
fc1_experts_bias (optional) : T
-
2D optional input tensor with shape (num_experts, inter_size)
+
fc2_experts_weights : T
-
3D input tensor with shape (num_experts, inter_size, hidden_size)
+
fc2_experts_bias (optional) : T
-
2D optional input tensor with shape (num_experts, hidden_size)
+
fc3_experts_weights (optional) : T
-
3D optional input tensor with shape (num_experts, hidden_size, inter_size)
+
fc3_experts_bias (optional) : T
-
2D optional input tensor with shape (num_experts, inter_size)
+
#### Outputs
output : T
-
2D input tensor with shape (num_rows, hidden_size) or 3D input tensor with shape (batch_size, sequence_length, hidden_size)
+
#### Type Constraints @@ -3131,15 +2712,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.MulInteger** - Performs element-wise binary quantized multiplication (with Numpy-style broadcasting support). - "This operator supports **multidirectional (i.e., Numpy-style) broadcasting**" - The output of this op is the int32 accumulated result of the mul operation - - ``` - C (int32) = (A - A_zero_point) * (B - B_zero_point) - ``` - - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3148,20 +2720,20 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T
-
First operand.
+
A_zero_point (optional) : T
-
Input A zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
B : T
-
Second operand.
+
B_zero_point (optional) : T
-
Input B zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
#### Outputs
C : T1
-
Constrain output to 32 bit tensor
+
#### Type Constraints @@ -3176,12 +2748,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.MultiHeadAttention** - Multi-Head Self/Cross Attention. Bias from input projection is included. - - The key padding mask is optional. When its shape is (batch_size, kv_sequence_length), value 0 - means padding or 1 otherwise. When key has right-side padding, its shape could be (batch_size): it is actual length of - each key sequence excluding paddings. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3203,32 +2769,32 @@ This version of the operator has been available since version 1 of the 'com.micr
query : T
-
Query with shape (batch_size, sequence_length, hidden_size), or packed QKV with shape (batch_size, kv_sequence_length, num_heads, 3, head_size)
+
key (optional) : T
-
Key with shape (batch_size, kv_sequence_length, hidden_size), or packed KV with shape (batch_size, kv_sequence_length, num_heads, 2, head_size), or past_key with shape (batch_size, num_heads, kv_sequence_length, head_size)
+
value (optional) : T
-
Value with shape (batch_size, kv_sequence_length, v_hidden_size), or past_value with shape (batch_size, num_heads, kv_sequence_length, head_size)
+
bias (optional) : T
-
Bias tensor with shape (hidden_size + hidden_size + v_hidden_size) from input projection
+
key_padding_mask (optional) : M
-
Key padding mask with shape (batch_size), (3 * batch_size + 2), (batch_size, kv_sequence_length), (batch_size, total_sequence_length), or (batch_size, sequence_length, total_sequence_length)
+
attention_bias (optional) : T
-
bias added to QxK' with shape (batch_size or 1, num_heads or 1, sequence_length, total_sequence_length)
+
past_key (optional) : T
-
past state for self attention key with shape (batch_size, num_heads, past_sequence_length, head_size)
+
past_value (optional) : T
-
past state for self attention value with shape (batch_size, num_heads, past_sequence_length, head_size)
+
#### Outputs (1 - 3)
output : T
-
3D output tensor with shape (batch_size, sequence_length, v_hidden_size)
+
present_key (optional) : T
-
present state for cross attention key with shape (batch_size, num_heads, kv_sequence_length, head_size)or present state for self attention key with shape (batch_size, num_heads, total_sequence_length, head_size)
+
present_value (optional) : T
-
present state for cross attention value with shape (batch_size, num_heads, kv_sequence_length, head_size)or present state for self attention value with shape (batch_size, num_heads, total_sequence_length, head_size)
+
#### Type Constraints @@ -3243,8 +2809,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.MurmurHash3** - The underlying implementation is MurmurHash3_x86_32 generating low latency 32bits hash suitable for implementing lookup tables, Bloom filters, count min sketch or feature hashing. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3262,14 +2826,14 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T1
-
An input tensor to hash.
+
#### Outputs
Y : T2
-
32-bit hash value.
+
#### Type Constraints @@ -3284,8 +2848,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.NGramRepeatBlock** - Enforce no repetition of n-grams. Scores are set to `-inf` for tokens that form a repeated n-gram if added to the back of the input_ids. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3301,16 +2863,16 @@ This version of the operator has been available since version 1 of the 'com.micr
input_ids : Tid
-
2D input tensor with shape (batch_size, sequence_length)
+
scores : T
-
2D input tensor with shape (batch_size, vocab_size)
+
#### Outputs
scores_out : T
-
2D output tensor with shape (batch_size, vocab_size)
+
#### Type Constraints @@ -3350,18 +2912,18 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
Input data tensor from previous layer; has size (N x C x H x W), where N is the batch size, C is the number of channels, and H and W are the height and width. Note that this is for the 2D image. Otherwise the size is (N x C x D1 x D2 ... x Dn). Optionally, if dimension denotation is in effect, the operation expects input data tensor to arrive with the dimension denotation of [DATA_BATCH, DATA_CHANNEL, DATA_FEATURE, DATA_FEATURE ...].
+
W : T
-
The weight tensor that will be used in the convolutions; has size (M x C/group x kH x kW), where C is the number of channels, and kH and kW are the height and width of the kernel, and M is the number of feature maps. For more than 2 dimensions, the kernel shape will be (M x C/group x k1 x k2 x ... x kn), where (k1 x k2 x ... kn) is the dimension of the kernel. Optionally, if dimension denotation is in effect, the operation expects the weight tensor to arrive with the dimension denotation of [FILTER_OUT_CHANNEL, FILTER_IN_CHANNEL, FILTER_SPATIAL, FILTER_SPATIAL ...]. Assuming zero based indices for the shape array, X.shape[1] == (W.shape[1] * group) == C and W.shape[0] mod G == 0. Or in other words FILTER_IN_CHANNEL multiplied by the number of groups should be equal to DATA_CHANNEL and the number of feature maps M should be a multiple of the number of groups G.
+
B (optional) : T
-
Optional 1D bias to be added to the convolution, has size of M.
+
#### Outputs
Y : T
-
Output data tensor that contains the result of the convolution. The output dimensions are functions of the kernel size, stride size, and pad lengths.
+
#### Type Constraints @@ -3374,9 +2936,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.NhwcFusedConv** - NhwcFusedConv is a Conv operator with optional activation and add operators fused in. - Only has fp16 implementation as of 2023/04/15. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3412,7 +2971,7 @@ This version of the operator has been available since version 1 of the 'com.micr
B (optional) : T
Z (optional) : T
-
Tensor to be added to the output, must be the same shape and format as the output tensor.
+
#### Outputs @@ -3477,27 +3036,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.PackedAttention** - This is the packed version of Attention. - - Sequences in one batch usually don't have same length and they are padded to have same length, - e.g., below is a batch with 3 sequences and tokens* are padded. - Sequence_0: 0, 1*, 2*, 3* - Sequence_1: 4, 5, 6*, 7* - Sequence_2: 8, 9, 10, 11 - - PackedAttention is designed to takes in packed input, i.e., only the real tokens without padding. - An input as above will be packed into 3 tensors like below: - - input ([h0, h4, h5, h8, h9, h10, h11]) - - token_offset: 0, 4, 5, 8, 9, 10, 11, 1*, 2*, 3*, 6*, 7* - - cumulated_token_count: 0, 1, 1+2, 1+2+4 - - Input tensors contains the hidden embedding of real tokens. - Token_offset records the offset of token in the unpacked input. - cumulated_token_count records cumulated length of each sequence length. - - The operator only supports BERT like model with padding on right now. - - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3517,24 +3055,24 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T
-
Input tensor with shape (token_count, input_hidden_size)
+
weights : T
-
Merged Q/K/V weights with shape (input_hidden_size, hidden_size + hidden_size + v_hidden_size)
+
bias : T
-
Bias tensor with shape (hidden_size + hidden_size + v_hidden_size) for input projection
+
token_offset : M
-
In packing mode, it specifies the offset of each token(batch_size, sequence_length).
+
cumulative_sequence_length : M
-
A tensor with shape (batch_size + 1). It specifies the cumulative sequence length.
+
attention_bias (optional) : T
-
A tensor with shape (batch_size or 1, num_heads or 1, sequence_length, sequence_length).It specifies the additional bias to QxK'
+
#### Outputs
output : T
-
2D output tensor with shape (token_count, v_hidden_size)
+
#### Type Constraints @@ -3549,28 +3087,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.PackedMultiHeadAttention** - This is the packed version of MultiHeadAttention. - - Sequences in one batch usually don't have same length and they are padded to have same length, - e.g., below is a batch with 3 sequences and * is padding token. - Sequence_0: 0, 1*, 2*, 3* - Sequence_1: 4, 5, 6*, 7* - Sequence_2: 8, 9, 10, 11 - - PackedMultiHeadAttention is designed to takes in packed input, i.e., only the real tokens without padding. - An input as above will be packed into 3 tensors like below: - - query ([q0, q4, q5, q8, q9, q10, q11]) - - key ([k0, k4, k5, k8, k9, k10, k11]) - - value ([v0, v4, v5, v8, v9, v10, v11]) - - token_offset: 0, 4, 5, 8, 9, 10, 11, 1*, 2*, 3*, 6*, 7* - - cumulative_sequence_length: 0, 1, 1+2, 1+2+4 - - The query, key and value tensors contain result of hidden embedding of real tokens after input projections. - Token_offset records the offset of token in the unpacked input. - cumulative_sequence_length records cumulated length of each sequence length. - - The operator only supports BERT like model with padding on right now. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3590,26 +3106,26 @@ This version of the operator has been available since version 1 of the 'com.micr
query : T
-
Query with shape (token_count, hidden_size) or packed qkv with shape (token_count, num_heads, 3, head_size)
+
key (optional) : T
-
Key with shape (token_count, hidden_size)
+
value (optional) : T
-
Value with shape (token_count, v_hidden_size)
+
bias (optional) : T
-
Bias tensor with shape (hidden_size + hidden_size + v_hidden_size) from input projection
+
token_offset : M
-
Offset of each token before packing, with shape (batch_size, sequence_length).
+
cumulative_sequence_length : M
-
A tensor with shape (batch_size + 1). It specifies the cumulative sequence length.
+
attention_bias (optional) : T
-
It specifies the additional bias to QxK'. The shape is (batch_size or 1, num_heads or 1, sequence_length, sequence_length)
+
#### Outputs
output : T
-
output tensor with shape (token_count, v_hidden_size)
+
#### Type Constraints @@ -3624,24 +3140,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.Pad** - Given `data` tensor, pads, mode, and value. - Example: - Insert 0 pads to the beginning of the second dimension. - data = [ - [1.0, 1.2], - [2.3, 3.4], - [4.5, 5.7], - ] - pads = [0, 2, 0, 0] - output = [ - [ - [0.0, 0.0, 1.0, 1.2], - [0.0, 0.0, 2.3, 3.4], - [0.0, 0.0, 4.5, 5.7], - ], - ] - - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3657,18 +3155,18 @@ This version of the operator has been available since version 1 of the 'com.micr
data : T
-
Input tensor.
+
pads : tensor(int64)
-
Tensor of integers indicating the number of padding elements to add or remove (if negative) at the beginning and end of each axis. For 2D input tensor, it is the number of pixels. `pads` should be a 1D tensor of shape [2 * input_rank] or a 2D tensor of shape [1, 2 * input_rank]. `pads` format (1D example) should be as follow [x1_begin, x2_begin,...,x1_end, x2_end,...], where xi_begin is the number of pixels added at the beginning of axis `i` and xi_end, the number of pixels added at the end of axis `i`.
+
value (optional) : T
-
(Optional) A scalar or rank 1 tensor containing a single value to be filled if the mode chosen is `constant` (by default it is 0.0).
+
#### Outputs
output : T
-
Tensor after padding.
+
#### Type Constraints @@ -3681,8 +3179,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QAttention** - Quantization of Multi-Head Self Attention. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3708,32 +3204,32 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T1
-
3D input tensor with shape (batch_size, sequence_length, input_hidden_size)
+
weight : T2
-
2D input tensor with shape (input_hidden_size, 3 * hidden_size), hidden_size = num_heads * head_size
+
bias : T3
-
1D input tensor with shape (3 * hidden_size)
+
input_scale : T3
-
scale of quantized input tensor. It's a scalar, which means a per-tensor/layer quantization.
+
weight_scale : T3
-
scale of weight scale. It's a scalar or a 1D tensor, which means a per-tensor/per-column quantization.Its size should be 3 * hidden_size if it is per-column quantization
+
mask_index (optional) : T4
-
Attention mask index with shape (batch_size)
+
input_zero_point (optional) : T1
-
zero point of quantized input tensor.It's a scalar, which means a per-tensor/layer quantization.
+
weight_zero_point (optional) : T2
-
zero point of quantized weight tensor. It's a scalar or a 1D tensor, which means a per-tensor/per-column quantization.Its size should be 3 * hidden_size if it is per-column quantization
+
past (optional) : T3
-
past state for key and value with shape (2, batch_size, num_heads, past_sequence_length, head_size).
+
#### Outputs (1 - 2)
output : T3
-
3D output tensor with shape (batch_size, sequence_length, hidden_size)
+
present (optional) : T3
-
present state for key and value with shape (2, batch_size, num_heads, past_sequence_length + sequence_length, head_size)
+
#### Type Constraints @@ -3752,8 +3248,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QGemm** - Quantized Gemm - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3773,30 +3267,30 @@ This version of the operator has been available since version 1 of the 'com.micr
A : TA
-
Input tensor A. The shape of A should be (M, K) if transA is 0, or (K, M) if transA is non-zero.
+
a_scale : T
-
Scale of quantized input 'A'. It is a scalar,which means a per-tensor quantization.
+
a_zero_point : TA
-
Zero point tensor for input 'A'. It is a scalar.
+
B : TB
-
Input tensor B. The shape of B should be (K, N) if transB is 0, or (N, K) if transB is non-zero.
+
b_scale : T
-
Scale of quantized input 'B'. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'B'.
+
b_zero_point : TB
-
Zero point tensor for input 'B'. It's optional and default value is 0. It could be a scalar or a 1-D tensor, which means a per-tensor or per-column quantization. If it's a 1-D tensor, its number of elements should be equal to the number of columns of input 'B'.
+
C (optional) : TC
-
Optional input tensor C. If not specified, the computation is done as if C is a scalar 0. The shape of C should be unidirectional broadcastable to (M, N). Its type is int32_t and must be quantized with zero_point = 0 and scale = alpha / beta * a_scale * b_scale.
+
y_scale (optional) : T
-
Scale of output 'Y'. It is a scalar, which means a per-tensor quantization. It is optional. The output is full precision(float32) if it is not provided. Or the output is quantized.
+
y_zero_point (optional) : TYZ
-
Zero point tensor for output 'Y'. It is a scalar, which means a per-tensor quantization. It is optional. The output is full precision(float32) if it is not provided. Or the output is quantized.
+
#### Outputs
Y : TY
-
Output tensor of shape (M, N).
+
#### Type Constraints @@ -3819,10 +3313,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QLinearAdd** - Performs element-wise binary addition on 8 bit data types (with Numpy-style broadcasting support). - - C = (A_scale * (A - A_zero_point) + B_scale * (B - B_zero_point))/C_scale + C_zero_point - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3831,28 +3321,28 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T
-
First operand.
+
A_scale : tensor(float)
-
Input A's scale. It's a scalar, which means a per-tensor/layer quantization.
+
A_zero_point (optional) : T
-
Input A zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
B : T
-
Second operand.
+
B_scale : tensor(float)
-
Input B's scale. It's a scalar, which means a per-tensor/layer quantization.
+
B_zero_point (optional) : T
-
Input B zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
C_scale : tensor(float)
-
Output scale. It's a scalar, which means a per-tensor/layer quantization.
+
C_zero_point (optional) : T
-
Output zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
#### Outputs
C : T
-
Result, has same element type as two inputs
+
#### Type Constraints @@ -3865,39 +3355,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QLinearAveragePool** - QLinearAveragePool consumes an input tensor X and applies average pooling across - the tensor according to kernel sizes, stride sizes, and pad lengths. - average pooling consisting of computing the average on all values of a - subset of the input tensor according to the kernel size and downsampling the - data into the output tensor Y for further processing. The output spatial shape will be following: - ``` - output_spatial_shape[i] = floor((input_spatial_shape[i] + pad_shape[i] - kernel_spatial_shape[i]) / strides_spatial_shape[i] + 1) - ``` - or - ``` - output_spatial_shape[i] = ceil((input_spatial_shape[i] + pad_shape[i] - kernel_spatial_shape[i]) / strides_spatial_shape[i] + 1) - ``` - if ceil_mode is enabled - - ``` - * pad_shape[i] is sum of pads along axis i - ``` - - `auto_pad` is a DEPRECATED attribute. If you are using them currently, the output spatial shape will be following: - ``` - VALID: output_spatial_shape[i] = ceil((input_spatial_shape[i] - kernel_spatial_shape[i] + 1) / strides_spatial_shape[i]) - SAME_UPPER or SAME_LOWER: output_spatial_shape[i] = ceil(input_spatial_shape[i] / strides_spatial_shape[i]) - ``` - And pad shape will be following if `SAME_UPPER` or `SAME_LOWER`: - ``` - pad_shape[i] = (output_spatial_shape[i] - 1) * strides_spatial_shape[i] + kernel_spatial_shape[i] - input_spatial_shape[i] - ``` - - The output of each pooling window is divided by the number of elements (exclude pad when attribute count_include_pad is zero). - - Input and output scales and zero points are used to convert the output to a new quantization range. - Output = Dequantize(Input) -> AveragePool on fp32 data -> Quantize(output) - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3925,22 +3382,22 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
Input data tensor from the previous operator; dimensions for image case are (N x C x H x W), where N is the batch size, C is the number of channels, and H and W are the height and the width of the data. For non image case, the dimensions are in the form of (N x C x D1 x D2 ... Dn), where N is the batch size. Optionally, if dimension denotation is in effect, the operation expects the input data tensor to arrive with the dimension denotation of [DATA_BATCH, DATA_CHANNEL, DATA_FEATURE, DATA_FEATURE ...].
+
x_scale : tensor(float)
-
Input scale. It's a scalar, which means a per-tensor/layer quantization.
+
x_zero_point (optional) : T
-
Input zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
y_scale : tensor(float)
-
Output scale. It's a scalar, which means a per-tensor/layer quantization.
+
y_zero_point (optional) : T
-
Output zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
#### Outputs
Y : T
-
Output data tensor from average or max pooling across the input tensor. Dimensions will vary based on various kernel, stride, and pad sizes. Floor value of the dimension is used
+
#### Type Constraints @@ -3953,8 +3410,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QLinearConcat** - Concatenate a list of tensors into a single tensor.All input tensors must have the same shape, except for the dimension size of the axis to concatenate on. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -3970,18 +3425,18 @@ This version of the operator has been available since version 1 of the 'com.micr
Y_scale : TF
-
Y's scale.
+
Y_zero_point : T8
-
Y's zero point.
+
inputs (variadic, heterogeneous) : TV
-
List of tensors/scale/zero_point for concatenation
+
#### Outputs
Y : T8
-
Concatenated tensor
+
#### Type Constraints @@ -4067,10 +3522,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QLinearGlobalAveragePool** - QLinearGlobalAveragePool consumes an input tensor X and applies Average pooling across - the values in the same channel. This is equivalent to AveragePool with kernel size - equal to the spatial dimension of input tensor. Input is of type uint8_t or int8_t. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4086,22 +3537,22 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
Input data tensor from the previous operator; According to channels_last, dimensions for image case are (N x C x H x W), or (N x H x W x C) where N is the batch size, C is the number of channels, and H and W are the height and the width of the data. For non image case, the dimensions are in the form of (N x C x D1 x D2 ... Dn), or (N x D1 X D2 ... Dn x C) where N is the batch size.
+
x_scale : tensor(float)
-
Scale of quantized input 'X'. It must be a scalar.
+
x_zero_point : T
-
Zero point tensor for input 'X'. It must be a scalar.
+
y_scale : tensor(float)
-
Scale of quantized output 'Y'. It must be a scalar.
+
y_zero_point : T
-
Zero point tensor for output 'Y'. It must be a scalar.
+
#### Outputs
Y : T
-
Output data tensor from pooling across the input tensor. The output tensor has the same rank as the input. with the N and C value keep it value, while the otherdimensions are all 1.
+
#### Type Constraints @@ -4114,10 +3565,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QLinearLeakyRelu** - QLinearLeakyRelu takes quantized input data (Tensor), an argument alpha, and quantize parameter for output, - and produces one output data (Tensor) where the function `f(x) = quantize(alpha * dequantize(x)) for dequantize(x) < 0`, - `f(x) = quantize(dequantize(x)) for dequantize(x) >= 0`, is applied to the data tensor elementwise. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4133,22 +3580,22 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
Input tensor
+
X_scale : tensor(float)
-
Input X's scale. It's a scalar, which means a per-tensor/layer quantization.
+
X_zero_point (optional) : T
-
Input X's zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
Y_scale : tensor(float)
-
Output Y's scale. It's a scalar, which means a per-tensor/layer quantization.
+
Y_zero_point (optional) : T
-
Output Y's zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
#### Outputs
Y : T
-
Output tensor
+
#### Type Constraints @@ -4161,10 +3608,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QLinearMul** - Performs element-wise binary multiplication on 8 bit data types (with Numpy-style broadcasting support). - - C = ((A - A_zero_point) * (B - B_zero_point)) * (A_scale * B_scale)/C_scale + C_zero_point - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4173,28 +3616,28 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T
-
First operand.
+
A_scale : tensor(float)
-
Input A's scale. It's a scalar, which means a per-tensor/layer quantization.
+
A_zero_point (optional) : T
-
Input A zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
B : T
-
Second operand.
+
B_scale : tensor(float)
-
Input B's scale. It's a scalar, which means a per-tensor/layer quantization.
+
B_zero_point (optional) : T
-
Input B zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
C_scale : tensor(float)
-
Output scale. It's a scalar, which means a per-tensor/layer quantization.
+
C_zero_point (optional) : T
-
Output zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
#### Outputs
C : T
-
Result, has same element type as two inputs
+
#### Type Constraints @@ -4207,18 +3650,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QLinearReduceMean** - Computes the mean of the low-precision input tensor's element along the provided axes. - The resulting tensor has the same rank as the input if keepdims equal 1. If keepdims equal 0, - then the resulting tensor have the reduced dimension pruned. The above behavior is similar to numpy, - with the exception that numpy default keepdims to False instead of True. - Input and Output scales and zero points are used to requantize the output in a new range. - This helps to improve accuracy as after ReduceMean operation the range of the output is expected to decrease. - - ``` - "Output = Dequantize(Input) -> ReduceMean on fp32 data -> Quantize(output)", - - ``` - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4236,22 +3667,22 @@ This version of the operator has been available since version 1 of the 'com.micr
data : T
-
An input tensor.
+
data_scale : tensor(float)
-
Input scale. It's a scalar, which means a per-tensor/layer quantization.
+
data_zero_point (optional) : T
-
Input zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
reduced_scale : tensor(float)
-
Output scale. It's a scalar, which means a per-tensor/layer quantization.
+
reduced_zero_point (optional) : T
-
Output zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
#### Outputs
reduced : T
-
Reduced output tensor.
+
#### Type Constraints @@ -4264,10 +3695,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QLinearSigmoid** - QLinearSigmoid takes quantized input data (Tensor), and quantize parameter for output, and produces one output data - (Tensor) where the function `f(x) = quantize(Sigmoid(dequantize(x)))`, is applied to the data tensor elementwise. - Wwhere the function `Sigmoid(x) = 1 / (1 + exp(-x))` - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4276,22 +3703,22 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
Input tensor
+
X_scale : tensor(float)
-
Input X's scale. It's a scalar, which means a per-tensor/layer quantization.
+
X_zero_point (optional) : T
-
Input X's zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
Y_scale : tensor(float)
-
Output Y's scale. It's a scalar, which means a per-tensor/layer quantization.
+
Y_zero_point (optional) : T
-
Output Y's zero point. Default value is 0 if it's not specified. It's a scalar, which means a per-tensor/layer quantization.
+
#### Outputs
Y : T
-
Output tensor
+
#### Type Constraints @@ -4304,13 +3731,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QLinearSoftmax** - QLinearSoftmax computes the normalized exponential values for the given input: - Softmax(input, axis) = Exp(input) / ReduceSum(Exp(input), axis=axis, keepdims=1) - The input does not need to explicitly be a 2D vector. The "axis" attribute - indicates the dimension along which QLinearSoftmax will be performed for onnx v.13+. - or the dimension coerced to NxD Matrix for onnx v.12-. - The output tensor has the same shape. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4328,22 +3748,22 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
The input tensor
+
X_scale : tensor(float)
-
Scale of quantized input 'X'. It must be a scalar.
+
x_zero_point (optional) : T
-
Zero point tensor for input 'X'.It must be a scalar.
+
y_scale : tensor(float)
-
Scale of quantized output 'Y'. It must be a scalar.
+
y_zero_point : T
-
Zero point tensor for output 'Y'. It must be a scalar.
+
#### Outputs
Y : T
-
Output data tensor from pooling across the input tensor. The output tensor has the same rank as the input.
+
#### Type Constraints @@ -4356,8 +3776,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QLinearWhere** - Return elements, either from X or Y, depending on condition. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4366,30 +3784,30 @@ This version of the operator has been available since version 1 of the 'com.micr
condition : B
-
When True (nonzero), yield x, otherwise yield y
+
X : T
-
Y's zero point.
+
x_scale : TF
-
X's scale.
+
x_zero_point : T
-
X's zero point.
+
Y : T
-
Y's zero point.
+
y_scale : TF
-
Y's scale.
+
y_zero_point : T
-
Y's zero point.
+
z_scale : TF
-
Z's scale.
+
z_zero_point : T
-
Z's zero point.
+
#### Outputs
Z : T
-
Tensor of shape equal to the broadcasted shape of condition, X, and Y
+
#### Type Constraints @@ -4406,8 +3824,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QMoE** - Quantized MoE - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4431,34 +3847,34 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T
-
2D input tensor with shape (num_rows, hidden_size) or 3D input tensor with shape (batch_size, sequence_length, hidden_size)
+
router_probs : T
-
2D input tensor with shape (num_rows, num_experts)
+
fc1_experts_weights : T1
-
3D input tensor with shape (num_experts, hidden_size, inter_size) or (num_experts, hidden_size, inter_size / 2)
+
fc1_scales : T
-
2D input tensor with shape (num_experts, inter_size)
+
fc1_experts_bias (optional) : T
-
2D optional input tensor with shape (num_experts, inter_size)
+
fc2_experts_weights : T1
-
3D input tensor with shape (num_experts, inter_size, hidden_size) or (num_experts, inter_size, hidden_size / 2)
+
fc2_scales : T
-
2D input tensor with shape (num_experts, hidden_size)
+
fc2_experts_bias (optional) : T
-
2D optional input tensor with shape (num_experts, hidden_size)
+
fc3_experts_weights (optional) : T1
-
3D optional input tensor with shape (num_experts, hidden_size, inter_size) or (num_experts, hidden_size, inter_size / 2)
+
fc3_scales (optional) : T
-
2D optional input tensor with shape (num_experts, inter_size)
+
fc3_experts_bias (optional) : T
-
2D optional input tensor with shape (num_experts, inter_size)
+
#### Outputs
output : T
-
2D input tensor with shape (num_rows, hidden_size) or 3D input tensor with shape (batch_size, sequence_length, hidden_size)
+
#### Type Constraints @@ -4473,18 +3889,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QOrderedAttention** - Quantized version of simplified Multi-Head Self Attention(using int8 with specific matrix Layout). - Multi-Head Self Attention that can be either unidirectional (like GPT-2) or bidirectional (like BERT). - The mask_index input is optional. Besides raw attention mask with shape (batch_size, past_sequence_length + sequence_length) - or (batch_size, sequence_length, past_sequence_length + sequence_length) with value 0 for masked and 1 otherwise, - we also support other two formats: When input has right-side padding, mask_index is one dimension with shape (batch_size), - where value of each element is the end position, or valid length of actual sequence excluding padding. When input has - left-side padding, mask_index has shape (2 * batch_size), where the values are the exclusive end positions followed by - the inclusive start positions. When unidirectional is 1, and each token only attend to previous tokens. For GPT-2, both past - and present state are optional. Present state could appear in output even when past state is not in input. - Current version does not support past/present, attention_bias and qkv_hidden_sizes. - TODO: Support them if needed in the future. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4510,52 +3914,52 @@ This version of the operator has been available since version 1 of the 'com.micr
input : Q
-
3D input tensor with shape (batch_size, sequence_length, input_hidden_size)
+
scale_input : S
-
scale of the input, scalar value (per tensor) currently.
+
scale_Q_gemm : S
-
scale of the gemm - scalar (per-tensor quantization)
+
scale_K_gemm : S
-
scale of the gemm - scalar (per-tensor quantization)
+
scale_V_gemm : S
-
scale of the gemm - scalar (per-tensor quantization)
+
Q_weight : Q
-
2D input tensor with shape (input_hidden_size, hidden_size), where hidden_size = num_heads * head_size
+
K_weight : Q
-
2D input tensor with shape (input_hidden_size, hidden_size), where hidden_size = num_heads * head_size
+
V_weight : Q
-
2D input tensor with shape (input_hidden_size, hidden_size), where hidden_size = num_heads * head_size
+
scale_Q_weight : S
-
scale of the weight (scalar for per-tensor quantization or 1-D of dims [hidden_size] for per-channel quantization)
+
scale_K_weight : S
-
scale of the weight (scalar for per-tensor quantization or 1-D of dims [hidden_size] for per-channel quantization)
+
scale_V_weight : S
-
scale of the weight (scalar for per-tensor quantization or 1-D of dims [hidden_size] for per-channel quantization)
+
Q_bias : S
-
1D input tensor with shape (hidden_size)
+
K_bias : S
-
1D input tensor with shape (hidden_size)
+
V_bias : S
-
1D input tensor with shape (hidden_size)
+
scale_QKT_gemm (optional) : S
-
scale of the gemm - scalar (per-tensor quantization)
+
scale_QKT_softmax (optional) : S
-
scale of the softmax result - scalar (per-tensor quantization)
+
scale_values_gemm : S
-
scale of the gemm - scalar (per-tensor quantization). Also this is the output scale for the operator.
+
mask_index (optional) : G
-
Attention mask with shape (batch_size, 1, max_sequence_length, max_sequence_length), (batch_size, past_sequence_length + sequence_length)or (batch_size, sequence_length, past_sequence_length + sequence_length), or index with shape (batch_size) or (2 * batch_size).
+
past (optional) : Q
-
past state for key and value with shape (2, batch_size, num_heads, past_sequence_length, head_size).
+
attention_bias (optional) : S
-
additional add to QxK' with shape (batch_size or 1, num_heads or 1, sequence_length, total_sequence_length).
+
#### Outputs
output : Q
-
3D output tensor with shape (batch_size, sequence_length, hidden_size)
+
#### Type Constraints @@ -4572,8 +3976,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QOrderedGelu** - Ordered Quantize Gelu. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4591,18 +3993,18 @@ This version of the operator has been available since version 1 of the 'com.micr
X : Q
-
N-dimensional input A
+
scale_X : S
-
scale of the input A
+
scale_Y : S
-
scale of the output Y
+
#### Outputs
Y : Q
-
Output of the Gelu
+
#### Type Constraints @@ -4617,8 +4019,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QOrderedLayerNormalization** - QOrderedLayerNormalization - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4640,22 +4040,22 @@ This version of the operator has been available since version 1 of the 'com.micr
X : Q
-
Input data tensor from the previous layer.
+
scale_X : S
-
scale of the quantized X
+
scale : F
-
Scale tensor, i.e., gamma vector.
+
B (optional) : F
-
Bias tensor.
+
scale_Y : S
-
scale of the quantized X
+
#### Outputs
Y : Q
-
Output data tensor.
+
#### Type Constraints @@ -4672,8 +4072,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QOrderedLongformerAttention** - Quantized version of Longformer Self Attention (using int8 with specific matrix Layout). - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4699,40 +4097,40 @@ This version of the operator has been available since version 1 of the 'com.micr
input : Q
-
3D input tensor with shape (batch_size, sequence_length, hidden_size), hidden_size = num_heads * head_size
+
scale_input : S
-
scale of the input
+
weight : Q
-
2D input tensor with shape (hidden_size, 3 * hidden_size)
+
scale_weight : S
-
scale of the weight
+
bias : S
-
1D input tensor with shape (3 * hidden_size), fp32 only currently.
+
scale_bias : S
-
reserved. (not used as add bias need float value in cublasLt for normal order.)
+
scale_qkv_gemm : S
-
scale of the output for fused kqv gemm
+
mask : F
-
Attention mask with shape (batch_size, sequence_length)
+
global_weight : Q
-
2D input tensor with shape (hidden_size, 3 * hidden_size)
+
scale_global_weight : S
-
scale of the global_weight
+
global_bias : S
-
1D input tensor with shape (3 * hidden_size)
+
scale_global_gemm : S
-
scale of the global_qkv_gemm
+
global : G
-
Global attention flags with shape (batch_size, sequence_length)
+
scale_output : S
-
scale of the output
+
#### Outputs
output : Q
-
3D output tensor with shape (batch_size, sequence_length, hidden_size)
+
#### Type Constraints @@ -4751,17 +4149,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QOrderedMatMul** - Quantize (Int8) MatMul with order. Implement Y = alpha * A * B + bias + beta * C. Matrix A, B, C, Y are all int8 matrix. - Two type of order combination supported: - *) When order_B is ORDER_COL, order_A must be ORDER_ROW. - bias is vector of {#cols of Y} of float32, C should be batch 1/batch_A. B could be of batch 1 or batch_A. - Note B is reorder to ORDER_COL, or Transposed. Not Transposed first and then Reordered here. - *) When order_B is specify ORDER_COL4_4R2_8C or ORDER_COL32_2R_4R4, orderA must be ORDER_COL32. - MatMul will be implemented using alpha(A * B) + beta * C => Y. - bias is not supported here. B in fact is transposed first then reordered into ORDER_COL4_4R2_8C or ORDER_COL32_2R_4R4 here. - order_Y and order_C will be same as order_A. - Support per column quantized weight, ie, scale_B is 1-D vector of size [#cols of matrix B]. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4781,28 +4168,28 @@ This version of the operator has been available since version 1 of the 'com.micr
A : Q
-
3-dimensional matrix A
+
scale_A : S
-
scale of the input A.
+
B : Q
-
2-dimensional matrix B. Transposed if order_B is ORDER_COL.
+
scale_B : S
-
scale of the input B. Scalar or 1-D float32.
+
scale_Y : S
-
scale of the output Y.
+
bias (optional) : S
-
1d bias, not scaled with scale_Y.
+
C (optional) : Q
-
3d or 2d matrix C. if 2d expand to 3d first. Shape[0] should be 1 or same as A.shape[0]
+
scale_C (optional) : S
-
scale of the input A.
+
#### Outputs
Y : Q
-
Matrix multiply results from A * B
+
#### Type Constraints @@ -4817,9 +4204,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QuantizeBFP** - The BFP quantization operator. It consumes a full precision tensor and computes an BFP tensor. - More documentation on the BFP format can be found in this paper: https://www.microsoft.com/en-us/research/publication/pushing-the-limits-of-narrow-precision-inferencing-at-cloud-scale-with-microsoft-floating-point/ - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4837,18 +4221,18 @@ This version of the operator has been available since version 1 of the 'com.micr
x : T1
-
N-D full precision input tensor to be quantized.
+
#### Outputs
y : T2
-
1-D, contiguous BFP data
+
shape : T3
-
Shape of x
+
strides : T3
-
Strides of x
+
#### Type Constraints @@ -4865,12 +4249,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QuantizeLinear** - The linear quantization operator. It consumes a full precision data, a scale, a zero point to compute the low precision / quantized tensor. - The quantization formula is y = saturate ((x / y_scale) + y_zero_point). For saturation, it saturates to [0, 255] if it's uint8, [-128, 127] if it's int8, - [0, 65,535] if it's uint16, and [-32,768, 32,767] if it's int16. For (x / y_scale), it's rounding to nearest ties to even. - Refer to https://en.wikipedia.org/wiki/Rounding for details. - Scale and zero point must have same shape. They must be either scalar (per tensor) or 1-D tensor (per 'axis'). - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4886,18 +4264,18 @@ This version of the operator has been available since version 1 of the 'com.micr
x : T1
-
N-D full precision Input tensor to be quantized.
+
y_scale : T1
-
Scale for doing quantization to get 'y'. It can be a scalar, which means per-tensor/layer quantization, or a 1-D tensor for per-axis quantization.
+
y_zero_point (optional) : T2
-
Zero point for doing quantization to get 'y'. Shape must match y_scale. Default is uint8 with zero point of 0 if it's not specified.
+
#### Outputs
y : T2
-
N-D quantized output tensor. It has same shape as input 'x'.
+
#### Type Constraints @@ -4912,8 +4290,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QuantizeWithOrder** - Quantize input matrix to specific layout used in cublaslt. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4931,16 +4307,16 @@ This version of the operator has been available since version 1 of the 'com.micr
input : F
-
TODO: input tensor of (ROWS, COLS). if less than 2d, will broadcast to (1, X). If 3d, it is treated as (B, ROWS, COS)
+
scale_input : S
-
scale of the input
+
#### Outputs
output : Q
-
output tensor
+
#### Type Constraints @@ -4957,8 +4333,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.QuickGelu** - Compute x * Sigmoid(alpha * x). - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -4974,14 +4348,14 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
The input data as Tensor.
+
#### Outputs
Y : T
-
The output.
+
#### Type Constraints @@ -4994,9 +4368,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.Range** - Creates a sequence of numbers that begins at `start` and extends by increments of `delta` - up to but not including `limit`. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5005,18 +4376,18 @@ This version of the operator has been available since version 1 of the 'com.micr
start : T
-
Tensor(scalar, or dims=[1]). First entry in the range.
+
limit : T
-
Tensor(scalar, or dims=[1]). Upper limit of sequence, exclusive.
+
delta (optional) : T
-
Tensor(scalar, or dims=[1]). Number that increments start. Defaults to 1.
+
#### Outputs
Y : T
-
1-D Tensor of the range.
+
#### Type Constraints @@ -5029,11 +4400,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.ReduceSumInteger** - Computes the sum of the low-precision input tensor's element along the provided axes. - The resulting tensor has the same rank as the input if keepdims equal 1. If keepdims equal 0, - then the resulting tensor have the reduced dimension pruned. The above behavior is similar to numpy, - with the exception that numpy default keepdims to False instead of True. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5051,14 +4417,14 @@ This version of the operator has been available since version 1 of the 'com.micr
data : T1
-
An input tensor.
+
#### Outputs
reduced : T2
-
Reduced output tensor.
+
#### Type Constraints @@ -5073,8 +4439,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.RelativePositionBias** - Compute binned relative position bias for T5 model. ref: https://arxiv.org/abs/1803.02155v2 - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5092,18 +4456,18 @@ This version of the operator has been available since version 1 of the 'com.micr
bias_table : T
-
2D input tensor with shape (num_buckets, num_heads), COL-major(See UT for example)
+
query_length : U
-
The length of query. Self Attention requires query_length = key_length
+
key_length : U
-
The length of key.
+
#### Outputs
output : T
-
4D output tensor with shape (1, num_heads, sequence_length, sequence_length)
+
#### Type Constraints @@ -5118,14 +4482,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.RemovePadding** - Compress transformer input by removing paddings. It assumes padding is on the right side of sequence. - - The input has padding with shape (batch_size, sequence_length, hidden_size). This will generate two outputs: - output has shape (total_tokens, hidden_size); token_offset with shape (batch_size, sequence_length). - - token_offset has offsets of all non-padding tokens first, then offset of all padding tokens. It is - a list of batch_size * sequence_length elements, which is reshaped to 2D for convenience of shape inference. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5134,22 +4490,22 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T
-
Input tensor with shape (batch_size, sequence_length, hidden_size)
+
sequence_token_count : M
-
Number of non-padding tokens in each sequence with shape (batch_size).
+
#### Outputs
output : T
-
output tensor with shape (total_tokens, hidden_size)
+
token_offset : M
-
Offset of non-padding tokens, and those of padding tokens. Its shape is (batch_size, sequence_length)
+
cumulated_seq_len : M
-
Cumulated sequence lengths. Its shape is (batch_size + 1)
+
max_seq_len : M
-
Max sequence length without padding. Its shape is (1)
+
#### Type Constraints @@ -5164,11 +4520,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.RestorePadding** - Restore paddings and fill padding with zeros. - - The input has padding with shape (total_tokens, hidden_size) and token_offset with shape (batch_size, sequence_length). - The output has shape (batch_size, sequence_length, hidden_size). - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5177,16 +4528,16 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T
-
Input tensor with shape (total_tokens, hidden_size)
+
token_offset : M
-
Offset of non-padding tokens and paddings. Its shape is (batch_size, sequence_length)
+
#### Outputs
output : T
-
output tensor with shape (batch_size, sequence_length, hidden_size)
+
#### Type Constraints @@ -5201,8 +4552,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.Rfft** - This function computes the n-point one dimensional Fourier transform for a real-valued input where n is an even number. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5222,14 +4571,14 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
input tensor of size n in the signal dim
+
#### Outputs
Y : T
-
output tensor of size (n//2 + 1) in the signal dim and 2 in the last dimension for the real and complex parts
+
#### Type Constraints @@ -5242,9 +4591,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.RotaryEmbedding** - RotaryEmbedding is the implementation of rotary positional embeddings (RoPE). The positions are represented as rotation matrices - that are multiplied to query and key before the inner product of query and key is taken. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5268,20 +4614,20 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T
-
3D tensor with shape (batch_size, sequence_length, hidden_size) or 4D with shape (batch_size, num_heads, sequence_length, head_size)
+
position_ids : M
-
1D tensor with shape (1) or 2D tensor with shape (batch_size, sequence_length)
+
cos_cache : T
-
2D tensor with shape (max_sequence_length, head_size / 2) or (max_sequence_length, rotary_embedding_dim / 2)
+
sin_cache : T
-
2D tensor with shape (max_sequence_length, head_size / 2) or (max_sequence_length, rotary_embedding_dim / 2)
+
#### Outputs
output : T
-
tensor with same shape as input.
+
#### Type Constraints @@ -5296,8 +4642,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.SampleOp** - Sample echo operator. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5306,14 +4650,14 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
input
+
#### Outputs
Y : T
-
output
+
#### Type Constraints @@ -5326,8 +4670,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.Sampling** - Greedy Sampling for text generation. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5371,32 +4713,32 @@ This version of the operator has been available since version 1 of the 'com.micr
input_ids : I
-
The sequence used as a prompt for the generation. Shape is (batch_size, sequence_length)
+
max_length : I
-
The maximum length of the sequence to be generated. Shape is (1)
+
min_length (optional) : I
-
The minimum length below which the score of eos_token_id is set to -Inf. Shape is (1)
+
repetition_penalty (optional) : T
-
The parameter for repetition penalty. Default value 1.0 means no penalty. Accepts value > 0.0. Shape is (1)
+
vocab_mask (optional) : I
-
Mask of vocabulary. Words that masked with 0 are not allowed to be generated, and 1 is allowed. Shape is (vocab_size)
+
prefix_vocab_mask (optional) : I
-
Mask of vocabulary for first step. Words that masked with 0 are not allowed to be generated, and 1 is allowed. Shape is (batch_size, vocab_size)
+
attention_mask (optional) : I
-
Custom attention mask. Shape is (batch_size, sequence_length)
+
presence_mask (optional) : I
-
Presence penalty mask. Shape is (batch_size, vocab_size)
+
seed (optional) : I
-
Seed for random number generator. Shape is (1)
+
#### Outputs (1 - 2)
sequences : I
-
Word IDs of generated sequences. Shape is (batch_size, max_sequence_length)
+
filtered_logits (optional) : T
-
Filtered logits as input to the mutinomial function for debug purpose. Shape is (batch_size, vocab_size)
+
#### Type Constraints @@ -5411,19 +4753,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.SkipGroupNorm** - This operator element-wise adds x, skip and bias, then apply group normalization and optional activation. - - This operator transforms input according to - s = x + skip + bias - y = gamma * (s - mean) / sqrt(variance + epsilon) + beta - - The input channels are separated into num_groups groups, each containing num_channels / num_groups channels. - The num_channels must be divisible by num_groups. - The mean and standard-deviation of s are calculated separately over the each group. - The weight and bias are per-channel affine transform parameter vectors of size num_channels. - - The activation attribute can be used to enable activation after group normalization. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5445,24 +4774,24 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
Input data tensor. Dimensions are (N x H x W x C) when channels_last is 1 or (N x C x H x W) otherwise, where N is the batch size, C is the number of channels, and H and W are the height and width of the data
+
gamma : M
-
1D gamma tensor for normalization with shape (C), where C is number of channels
+
beta : M
-
1D beta tensor for normalization with shape (C), where C is number of channels
+
skip : T
-
4D or 2D skip tensor. The shape can be (N x H x W x C) or (N x 1 x 1 x C) or (N x C)
+
bias (optional) : T
-
1D bias tensor. Dimensions are (C), where C is number of channels
+
#### Outputs (1 - 2)
Y : T
-
The output tensor of the same shape as X
+
S (optional) : T
-
The element-wise sum of input x, skip and bias tensors. It has the same shape as X
+
#### Type Constraints @@ -5477,8 +4806,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.SkipLayerNormalization** - Skip and Layer Normalization Fusion - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5494,28 +4821,28 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T
-
3D input tensor with shape (batch_size, sequence_length, hidden_size)
+
skip : T
-
3D skip tensor with shape (batch_size, sequence_length, hidden_size) or (1, sequence_length, hidden_size) or (sequence_length, hidden_size)
+
gamma : T
-
1D input tensor with shape (hidden_size)
+
beta (optional) : T
-
1D skip tensor with shape (hidden_size
+
bias (optional) : T
-
1D bias tensor with shape (hidden_size
+
#### Outputs (1 - 4)
output : T
-
3D output tensor with shape (batch_size, sequence_length, hidden_size)
+
mean (optional) : U
-
Saved mean used during training to speed up gradient computation
+
inv_std_var (optional) : U
-
Saved inverse standard variance used during training to speed up gradient computation.
+
input_skip_bias_sum (optional) : T
-
Sum of the input and skip inputs (and bias if it exists) with shape (batch_size, sequence_length, hidden_size).
+
#### Type Constraints @@ -5530,8 +4857,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.SkipSimplifiedLayerNormalization** - Skip and Root Mean Square Layer Normalization - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5547,26 +4872,26 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T
-
3D input tensor with shape (batch_size, sequence_length, hidden_size)Or 2D input tensor with shape (token_count, hidden_size)
+
skip : T
-
3D input tensor with shape (batch_size, sequence_length, hidden_size)Or 2D input tensor with shape (token_count, hidden_size)
+
gamma : T
-
1D input tensor with shape (hidden_size)
+
bias (optional) : T
-
1D bias tensor with shape (hidden_size
+
#### Outputs (1 - 4)
output : T
-
3D output tensor with shape (batch_size, sequence_length, hidden_size)Or 2D output tensor with shape (token_count, hidden_size)
+
mean (optional) : U
-
Saved mean used during training to speed up gradient computation
+
inv_std_var (optional) : U
-
Saved inverse standard variance used during training to speed up gradient computation.
+
input_skip_bias_sum (optional) : T
-
Sum of the input and skip inputs (and bias if it exists)with shape (batch_size, sequence_length, hidden_size) or (token_count, hidden_size).
+
#### Type Constraints @@ -5581,8 +4906,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.Snpe** - Onnx node for SNPE. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5604,14 +4927,14 @@ This version of the operator has been available since version 1 of the 'com.micr
inputs (variadic) : T
-
List of tensors for SNPE DLC input
+
#### Outputs (1 - ∞)
outputs (variadic) : T
-
One or more outputs, list of tensors for DLC output
+
#### Type Constraints @@ -5624,39 +4947,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.SparseAttention** - Block Sparse Attention used in Phi-3-small (https://arxiv.org/pdf/2404.14219). - - It is inspired by Sparse Transformers (https://arxiv.org/pdf/1904.10509) and BigBird (https://arxiv.org/pdf/2007.14062). - - block_mask can be used to configure sparse layout for different head. - When number of sparse layout is 1, all heads have same sparse layout. Otherwise, different layouts are used cyclically. - For example, given 4 layouts (S0, S1, S2, S3), 8 heads will have layouts like (S0, S1, S2, S3, S0, S1, S2, S3). - - The block_row_indices and block_col_indices are the CSR representation of block mask. The block_col_indices might contain - paddings at the right side when different layout has different number of non-zeros in block mask. - - An example of block mask with 2 layouts where each layout is 4 x 4 blocks: - [[[1, 0, 0, 0], - [1, 1, 0, 0], - [0, 1, 1, 0], - [0, 1, 1, 1]], - - [[1, 0, 0, 0], - [1, 1, 0, 0], - [1, 1, 1, 0], - [1, 0, 1, 1]]] - - The corresponding CSR format: - block_col_indices = [[0, 0, 1, 1, 2, 1, 2, 3, -1], [0, 0, 1, 0, 1, 2, 0, 2, 3]] - block_row_indices = [[0, 1, 3, 5, 8], [0, 1, 3, 6, 9]] - - When do_rotary is True, cos_cache and sin_cache are required. Note that the maximum sequence length supported by cos - or sin cache can be different from the maximum sequence length used by kv cache. - - Only supports unidirectional attention with cache of past key and value in linear buffers. - - For performance, past_key and present_key share same memory buffer, and past_value and present_value too. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5682,38 +4972,38 @@ This version of the operator has been available since version 1 of the 'com.micr
query : T
-
Query with shape (batch_size, sequence_length, num_heads * head_size), or packed QKV with shape is(batch_size, sequence_length, d) where d is (num_heads + 2 * kv_num_heads) * head_size.
+
key (optional) : T
-
Key with shape (batch_size, sequence_length, kv_num_heads * head_size)
+
value (optional) : T
-
Value with shape (batch_size, sequence_length, kv_num_heads * head_size)
+
past_key : T
-
Key cache with shape (batch_size, kv_num_heads, max_cache_sequence_length, head_size)
+
past_value : T
-
Value cache with shape (batch_size, kv_num_heads, max_cache_sequence_length, head_size)
+
block_row_indices : M
-
The row indices of CSR format of block mask with shape (num_layout, max_blocks + 1).The num_heads is divisible by num_layout, and max_blocks is max_sequence_length / sparse_block_size.
+
block_col_indices : M
-
The col indices of CSR format of block mask with shape (num_layout, max_nnz_blocks).The max_nnz_blocks is the maximum number of non-zeros per layout in block mask.
+
total_sequence_length : M
-
Scalar tensor of maximum total sequence length (past_sequence_length + sequence_length) among keys.
+
key_total_sequence_lengths : M
-
1D tensor with shape (batch_size) where each value is total sequence length of key excluding paddings.
+
cos_cache (optional) : T
-
Cos cache of rotary with shape (max_rotary_sequence_length, head_size / 2).
+
sin_cache (optional) : T
-
Sin cache of rotary with shape (max_rotary_sequence_length, head_size / 2).
+
#### Outputs
output : T
-
3D output tensor with shape (batch_size, sequence_length, num_heads * head_size)
+
present_key : T
-
Updated key cache with shape (batch_size, kv_num_heads, max_cache_sequence_length, head_size).
+
present_value : T
-
Updated value cache with shape (batch_size, kv_num_heads, max_cache_sequence_length, head_size).
+
#### Type Constraints @@ -5747,16 +5037,16 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T
-
2-dimensional sparse matrix A. Either COO or CSR format
+
B : T1
-
N-dimensional dense matrix B
+
#### Outputs
Y : T1
-
Matrix multiply results
+
#### Type Constraints @@ -5771,37 +5061,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.Tokenizer** - Tokenizer divides each string in X into a vector of strings along the last axis. Allowed input shapes are [C] and [N, C]. - If the maximum number of tokens found per input string is D, the output shape would be [N, C, D] when input shape is [N, C]. - Similarly, if input shape is [C] then the output should be [C, D]. Tokenizer has two different operation modes. - The first mode is selected when "tokenexp" is not set and "separators" is set. If "tokenexp" is set and "separators" is not set, - the second mode will be used. The first mode breaks each input string into tokens by matching and removing separators. - "separators" is a list of strings which are regular expressions. "tokenexp" is a single regular expression. - Let's assume "separators" is [" "] and consider an example. - If input is - ["Hello World", "I love computer science !"] whose shape is [2], - then the output would be - [["Hello", "World", padvalue, padvalue, padvalue], - ["I", "love", "computer", "science", "!"]] - whose shape is [2, 5] because you can find at most 5 tokens per input string. - Note that the input at most can have two axes, so 3-D and higher dimension are not supported. - If "separators" contains a single empty string, the Tokenizer will enter into character tokenezation mode. This means all strings - will be broken part into individual characters. - For each input string, the second mode searches matches of "tokenexp" and each match will be a token in Y. - The matching of "tokenexp" is conducted greedily (i.e., a match should be as long as possible). - This operator searches for the first match starting from the beginning of the considered string, - and then launches another search starting from the first remained character after the first matched token. - If no match found, this operator will remove the first character from the remained string and do another search. - This procedure will be repeated until reaching the end of the considered string. - Let's consider another example to illustrate the effect of setting "mark" to true. - If input is ["Hello", "World"], - then the corresponding output would be [0x02, "Hello", "World", 0x03]. - This implies that if mark is true, [C]/[N, C] - input's output shape becomes [C, D+2]/[N, C, D+2]. - If tokenizer removes the entire content of [C]-input, it will produce [[]]. - I.e. the output shape should be [C][0] or [N][C][0] if input shape was [N][C]. - If the tokenizer receives empty input of [0] then the output is [0] if empty input - of [N, 0] then [N, 0]. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5825,14 +5084,14 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
Strings to tokenize
+
#### Outputs
Y : T
-
Tokenized strings
+
#### Type Constraints @@ -5845,10 +5104,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.TorchEmbedding** - Based on Torch operator Embedding, creates a lookup table of embedding vectors of fixed size, - for a dictionary of fixed size. - - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5857,20 +5112,20 @@ This version of the operator has been available since version 1 of the 'com.micr
weight : T
-
The embedding matrix of size N x M. 'N' is equal to the maximum possible index + 1, and 'M' is equal to the embedding size
+
indices : tensor(int64)
-
Long tensor containing the indices to extract from embedding matrix.
+
padding_idx (optional) : tensor(int64)
-
A 0-D scalar tensor. If specified, the entries at `padding_idx` do not contribute to the gradient; therefore, the embedding vector at `padding_idx` is not updated during training, i.e. it remains as a fixed pad.
+
scale_grad_by_freq (optional) : tensor(bool)
-
A 0-D bool tensor. If given, this will scale gradients by the inverse of frequency of the indices (words) in the mini-batch. Default is ``False``
+
#### Outputs
Y : T
-
Output tensor of the same type as the input tensor. Shape of the output is * x M, where '*' is the shape of input indices, and 'M' is the embedding size.
+
#### Type Constraints @@ -5883,9 +5138,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.TransposeMatMul** - Duplicate of FusedMatMul. Going forward FusedMatMul should be used. This OP will be supported for backward compatibility. - Matrix product that behaves like numpy.matmul: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.matmul.html - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5905,16 +5157,16 @@ This version of the operator has been available since version 1 of the 'com.micr
A : T
-
N-dimensional matrix A
+
B : T
-
N-dimensional matrix B
+
#### Outputs
Y : T
-
Matrix multiply results
+
#### Type Constraints @@ -5927,18 +5179,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.Trilu** - Returns the upper or lower triangular part of a 2-D matrix, or batches of 2-D matrices. If the attribute "upper" is set to true, - the upper triangular matrix is retained. Lower triangular matrix is retained otherwise. Default value for upper is true. - Trilu takes one input tensor of shape [*, N, M], where * is zero or more batch dimensions. The upper triangular part consists - of the elements on and above the given diagonal (k). The lower triangular part consists of elements on and below the diagonal. - All other elements in the matrix are set to zero. - If k = 0, the triangular part on and above/below the main diagonal is retained. - If upper is set to true, a positive k retains the upper triangular matrix excluding k diagonals above - the main diagonal. A negative k value includes as many diagonals below the main diagonal. - If upper is set to false, a positive k retains the lower triangular matrix including k diagonals above - the main diagonal. A negative k value excludes as many diagonals below the main diagonal. - - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5954,16 +5194,16 @@ This version of the operator has been available since version 1 of the 'com.micr
X : T
-
Input tensor of rank 2 or higher.
+
k (optional) : tensor(int64)
-
A 0-D tensor containing a single value corresponding to the number diagonals above or the main diagonal to exclude or include.Default value is 0 if it's not specified.
+
#### Outputs
Y : T
-
Output tensor of the same type and shape as the input tensor.
+
#### Type Constraints @@ -5976,8 +5216,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.UnfoldTensor** - Returns a tensor which contains all slices of size `size` from input tensor in the dimension `dim`. Step between two slices is given by `step`. If `sizedim` is the size of dimension `dim` for input tensor, the size of dimension `dim` in the returned tensor will be `(sizedim - size) / step + 1`. An additional dimension of size `size` is appended in the returned tensor. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -5997,14 +5235,14 @@ This version of the operator has been available since version 1 of the 'com.micr
input : T
-
input tensor
+
#### Outputs
output : T
-
Output tensor.
+
#### Type Constraints @@ -6017,20 +5255,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.Unique** - Finds all the unique values (deduped list) present in the given input tensor. - This operator returns 3 outputs. - The first output tensor 'uniques' contains all of the unique elements of the input, - sorted in the same order that they occur in the input. - The second output tensor 'idx' is the same size as the input and it contains the index - of each value of the input in 'uniques'. - The third output tensor 'counts' contains the count of each element of 'uniques' in the input. - Example: - input_x = [2, 1, 1, 3, 4, 3] - output_uniques = [2, 1, 3, 4] - output_idx = [0, 1, 1, 2, 3, 2] - output_counts = [1, 2, 2, 1] - - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -6039,18 +5263,18 @@ This version of the operator has been available since version 1 of the 'com.micr
x : T
-
A 1-D input tensor that is to be processed.
+
#### Outputs
y : T
-
A 1-D tensor of the same type as 'x' containing all the unique values in 'x' sorted in the same order that they occur in the input 'x'
+
idx : tensor(int64)
-
A 1-D INT64 tensor of the same size as 'x' containing the indices for each value in 'x' in the output 'uniques'
+
counts : tensor(int64)
-
A 1-D INT64 tensor containing the the count of each element of 'uniques' in the input 'x'
+
#### Type Constraints @@ -6063,8 +5287,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.WhisperBeamSearch** - Beam Search for whisper model, especiall with cross_qk features etc. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -6112,50 +5334,50 @@ This version of the operator has been available since version 1 of the 'com.micr
input_ids : F
-
The sequence used as a prompt for the generation in the encoder subgraph. Shape is (batch_size, sequence_length)
+
max_length : I
-
The maximum length of the sequence to be generated. Shape is (1)
+
min_length (optional) : I
-
The minimum length below which the score of eos_token_id is set to -Inf. Shape is (1)
+
num_beams : I
-
Number of beams for beam search. 1 means no beam search. Shape is (1)
+
num_return_sequences : I
-
The number of returned sequences in the batch. Shape is (1)
+
length_penalty (optional) : T
-
Exponential penalty to the length. Default value 1.0 means no penalty. Value > 1.0 encourages longer sequences, while values < 1.0 produces shorter sequences. Shape is (1,)
+
repetition_penalty (optional) : T
-
The parameter for repetition penalty. Default value 1.0 means no penalty. Accepts value > 0.0. Shape is (1)
+
vocab_mask (optional) : M
-
Mask of vocabulary. Words that masked with 0 are not allowed to be generated, and 1 is allowed. Shape is (vocab_size)
+
prefix_vocab_mask (optional) : M
-
Mask of vocabulary for first step. Words that masked with 0 are not allowed to be generated, and 1 is allowed. Shape is (batch_size, vocab_size)
+
attention_mask (optional) : I
-
Custom attention mask. Shape is (batch_size, sequence_length)
+
decoder_input_ids (optional) : I
-
The forced input id sequence for the decoder subgraph. Shape is (batch_size, initial_sequence_length)
+
logits_processor (optional) : I
-
Specific logits processor for different types of beamsearch models. Default value 0 means no specific logit processor. Accepts value >= 0. Shape is (1)
+
cross_qk_layer_head (optional) : I
-
Only keep this list of (layer, head) of QK in the final cross_qk output when use_cross_qk is set. Default collect all its shape is (number of (layer, head) to keep, 2), i.e., [[layer_id1, head_id1], [layer_id2, head_id2]......]
+
extra_decoding_ids (optional) : I
-
Part of the decoder_input_ids that we need cross qk for it. it is of shape (batch_size, extra_decoding_ids_len).In such case, we should remove this from the tail of the decoder_input_ids, and put it here. ids < 0 in it (for multiple batch) are treated as stop of the extra_decoding_ids for corresponding batch.
+
temperature (optional) : T
-
Temperature value to apply to logits processing during this execution's decoding. Shape is (1)
+
#### Outputs (1 - 5)
sequences : I
-
Word IDs of generated sequences. Shape is (batch_size, num_return_sequences, max_sequence_length)
+
sequences_scores (optional) : T
-
Final beam score of the generated sequences. Shape is (batch_size, num_return_sequences)
+
scores (optional) : T
-
Processed beam scores for each vocabulary token at each generation step. Beam scores consisting of log softmax scores for each vocabulary token and sum of log softmax of previously generated tokens in this beam. Shape is (max_length - sequence_length, batch_size, num_beams, vocab_size)
+
cross_qk (optional) : V
-
Output the accumulated stacked Q*K in cross attentions. Let H = number of Head of cross attention, F = the frames or kv-seq-len of the cross attention input, T = real decoded token length, L = number of layers, B = batch size, R = num_return_sequences. It then should return tensor of shape [B, R, L*H, T, F]. If cross_qk_layer_head is given, shape is [B, R, cross_qk_layer_head.shape[0], T, F]
+
non_speech_probs (optional) : T
-
For whisper model, output the probabilities from logits after encoder and context decoding for the no_speech_token_id. The shape of non_speech_probs is [B]
+
#### Type Constraints @@ -6176,8 +5398,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### **com.microsoft.WordConvEmbedding** - The WordConvEmbedding takes in a batch of sequence words and embed each word to a vector. - #### Version This version of the operator has been available since version 1 of the 'com.microsoft' operator set. @@ -6197,20 +5417,20 @@ This version of the operator has been available since version 1 of the 'com.micr
Sequence : T
-
Specify batchs of sequence words to embedding
+
W : T1
-
Specify weights of conv
+
B : T1
-
Specify bias of conv
+
C : T1
-
Specify embedding vector of char
+
#### Outputs
Y : T1
-
output
+
#### Type Constraints @@ -6225,8 +5445,6 @@ This version of the operator has been available since version 1 of the 'com.micr ### experimental **com.microsoft.IsAllFinite** - IsAllFinite - #### Version No versioning maintained for experimental ops. @@ -6243,14 +5461,14 @@ No versioning maintained for experimental ops.
input (variadic) : V
-
Input tensors to check.
+
#### Outputs
output : T
-
The output scalar. Its value is true if all input tensors are finite. Otherwise, the output value would be false.
+
#### Type Constraints @@ -6265,12 +5483,6 @@ No versioning maintained for experimental ops. ### experimental **com.microsoft.QEmbedLayerNormalization** - QEmbedLayerNormalization is the quantized fusion of embedding layer in BERT model, with optional mask processing. - The embedding layer takes input_ids (word IDs) and segment_ids (sentence IDs) to look up word_embedding, position_embedding, - and segment_emedding; the embeddings are added then applied layer normalization using gamma and beta tensors. The input_ids - and segment_ids remain int32. All embeddings, gamma, and beta tensors are converted to int8/uint8. The last input mask is optional. - If mask is provided, mask index (that is position of first 0 in mask, or number of words will be calculated. - #### Version No versioning maintained for experimental ops. @@ -6285,50 +5497,50 @@ No versioning maintained for experimental ops.
input_ids : T1
-
2D words IDs with shape (batch_size, sequence_length)
+
segment_ids (optional) : T1
-
2D segment IDs with shape (batch_size, sequence_length)
+
word_embedding_quant : T2
-
2D with shape (,hidden_size)
+
position_embedding_quant : T2
-
2D with shape (, hidden_size)
+
segment_embedding (optional) : T2
-
2D with shape (, hidden_size)
+
gamma_quant : T2
-
1D gamma tensor for layer normalization with shape (hidden_size)
+
beta_quant : T2
-
1D beta tensor for layer normalization with shape (hidden_size)
+
mask (optional) : T1
-
Mask
+
word_embedding_scale : T
-
Scale for word embeddings
+
position_embedding_scale : T
-
Scale for position embeddings
+
segment_embedding_scale (optional) : T
-
Scale for segment embeddings
+
gamma_scale : T
-
Scale for 1D gamma tensor
+
beta_scale : T
-
Scale for 1D beta tensor
+
word_embedding_zero_point : T2
-
Zero point for word embeddings
+
position_embedding_zero_point : T2
-
Zero point for position embeddings
+
segment_embedding_zero_point (optional) : T2
-
Zero Point for segment embeddings
+
gamma_zero_point : T2
-
Zero Point for 1D gamma tensor
+
beta_zero_point : T2
-
Zero Point for 1D beta tensor
+
#### Outputs
layernorm_out : T
-
LayerNorm Output
+
mask_index_out : T1
-
Mask Index Output
+
#### Type Constraints @@ -6343,3 +5555,937 @@ No versioning maintained for experimental ops. +## com.microsoft.nchwc +### **com.microsoft.nchwc.AveragePool** + +#### Version + +This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set. + +#### Attributes + +
+
auto_pad : string
+
+
ceil_mode : int
+
+
count_include_pad : int
+
+
dilations : list of ints
+
+
kernel_shape : list of ints (required)
+
+
pads : list of ints
+
+
strides : list of ints
+
+
+ +#### Inputs + +
+
X : T
+
+
+ +#### Outputs + +
+
Y : T
+
+
+ +#### Type Constraints + +
+
T : tensor(float)
+
Constrain input and output types to float tensors
+
+ + +### **com.microsoft.nchwc.Conv** + +#### Version + +This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set. + +#### Attributes + +
+
activation : string
+
+
activation_params : list of floats
+
+
auto_pad : string
+
+
dilations : list of ints
+
+
group : int
+
+
kernel_shape : list of ints
+
+
pads : list of ints
+
+
strides : list of ints
+
+
+ +#### Inputs (2 - 4) + +
+
X : T
+
+
W : T
+
+
B (optional) : T
+
+
Sum (optional) : T
+
+
+ +#### Outputs + +
+
Y : T
+
+
+ +#### Type Constraints + +
+
T : tensor(float)
+
Constrain input and output types to float tensors
+
+ + +### **com.microsoft.nchwc.GlobalAveragePool** + +#### Version + +This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set. + +#### Inputs + +
+
X : T
+
+
+ +#### Outputs + +
+
Y : T
+
+
+ +#### Type Constraints + +
+
T : tensor(float)
+
Constrain input and output types to float tensors
+
+ + +### **com.microsoft.nchwc.GlobalMaxPool** + +#### Version + +This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set. + +#### Inputs + +
+
X : T
+
+
+ +#### Outputs + +
+
Y : T
+
+
+ +#### Type Constraints + +
+
T : tensor(float)
+
Constrain input and output types to float tensors
+
+ + +### **com.microsoft.nchwc.MaxPool** + +#### Version + +This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set. + +#### Attributes + +
+
auto_pad : string
+
+
ceil_mode : int
+
+
dilations : list of ints
+
+
kernel_shape : list of ints (required)
+
+
pads : list of ints
+
+
storage_order : int
+
+
strides : list of ints
+
+
+ +#### Inputs + +
+
X : T
+
+
+ +#### Outputs + +
+
Y : T
+
+
+ +#### Type Constraints + +
+
T : tensor(float)
+
Constrain input and output types to float tensors
+
+ + +### **com.microsoft.nchwc.ReorderInput** + +#### Version + +This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set. + +#### Attributes + +
+
channels_last : int
+
+
+ +#### Inputs + +
+
X : T
+
+
+ +#### Outputs + +
+
Y : T
+
+
+ +#### Type Constraints + +
+
T : tensor(float)
+
Constrain input and output types to float tensors
+
+ + +### **com.microsoft.nchwc.ReorderOutput** + +#### Version + +This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set. + +#### Attributes + +
+
channels : int
+
+
channels_last : int
+
+
+ +#### Inputs + +
+
X : T
+
+
+ +#### Outputs + +
+
Y : T
+
+
+ +#### Type Constraints + +
+
T : tensor(float)
+
Constrain input and output types to float tensors
+
+ + +### **com.microsoft.nchwc.Upsample** + +#### Version + +This version of the operator has been available since version 1 of the 'com.microsoft.nchwc' operator set. + +#### Attributes + +
+
coordinate_transformation_mode : string
+
+
mode : string
+
+
scales : list of ints
+
+
+ +#### Inputs + +
+
X : T
+
+
+ +#### Outputs + +
+
Y : T
+
+
+ +#### Type Constraints + +
+
T : tensor(float)
+
Constrain input and output types to float tensors
+
+ + +## com.ms.internal.nhwc +### **com.ms.internal.nhwc.BatchNormalization** + +#### Version + +This version of the operator has been available since version 15 of the 'com.ms.internal.nhwc' operator set. + +Other versions of this operator: com.ms.internal.nhwc.BatchNormalization-7, com.ms.internal.nhwc.BatchNormalization-9, com.ms.internal.nhwc.BatchNormalization-14 + +#### Attributes + +
+
activation : string
+
+
activation_params : list of floats
+
+
epsilon : float
+
The epsilon value to use to avoid division by zero.
+
momentum : float
+
Factor used in computing the running mean and variance.e.g., running_mean = running_mean * momentum + mean * (1 - momentum).
+
training_mode : int
+
If set to true, it indicates BatchNormalization is being used for training, and outputs 1 and 2 are to be computed.
+
+ +#### Inputs + +
+
X : T
+
+
scale : T1
+
+
B : T1
+
+
input_mean : T2
+
+
input_var : T2
+
+
+ +#### Outputs (1 - 3) + +
+
Y : T
+
+
running_mean (optional) : T2
+
+
running_var (optional) : T2
+
+
+ +#### Type Constraints + +
+
T : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
+
Constrain input and output types to float tensors.
+
T1 : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
+
Constrain scale and bias types to float tensors.
+
T2 : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
+
Constrain mean and variance types to float tensors.
+
+ + +### **com.ms.internal.nhwc.ConvTranspose** + +#### Version + +This version of the operator has been available since version 11 of the 'com.ms.internal.nhwc' operator set. + +Other versions of this operator: com.ms.internal.nhwc.ConvTranspose-1 + +#### Attributes + +
+
activation : string
+
+
activation_params : list of floats
+
+
auto_pad : string
+
auto_pad must be either NOTSET, SAME_UPPER, SAME_LOWER or VALID. Where default value is NOTSET, which means explicit padding is used. SAME_UPPER or SAME_LOWER mean pad the input so that `output_shape[i] = input_shape[i] * strides[i]` for each axis `i`. The padding is split between the two sides equally or almost equally (depending on whether it is even or odd). In case the padding is an odd number, the extra padding is added at the end for SAME_UPPER and at the beginning for SAME_LOWER.
+
dilations : list of ints
+
dilation value along each spatial axis of the filter. If not present, the dilation defaults to 1 along each spatial axis.
+
group : int
+
number of groups input channels and output channels are divided into.
+
kernel_shape : list of ints
+
The shape of the convolution kernel. If not present, should be inferred from input W.
+
output_padding : list of ints
+
Additional elements added to the side with higher coordinate indices in the output. Each padding value in "output_padding" must be less than the corresponding stride/dilation dimension. By default, this attribute is a zero vector. Note that this attribute doesn't directly affect the computed output values. It only controls the selection of the computed values, so changing this attribute only adds or removes output elements. If "output_shape" is explicitly provided, "output_padding" does not contribute additional size to "output_shape" but participates in the computation of the needed padding amount. This is also called adjs or adjustment in some frameworks.
+
output_shape : list of ints
+
The shape of the output can be explicitly set which will cause pads values to be auto generated. If output_shape is specified pads values are ignored. See doc for details for equations to generate pads. Note that the output_shape attribute value should not include dimensions for batch size and channels, which are automatically inferred.
+
pads : list of ints
+
Padding for the beginning and ending along each spatial axis, it can take any value greater than or equal to 0. The value represent the number of pixels added to the beginning and end part of the corresponding axis. `pads` format should be as follow [x1_begin, x2_begin...x1_end, x2_end,...], where xi_begin the number of pixels added at the beginning of axis `i` and xi_end, the number of pixels added at the end of axis `i`. This attribute cannot be used simultaneously with auto_pad attribute. If not present, the padding defaults to 0 along start and end of each spatial axis.
+
strides : list of ints
+
Stride along each spatial axis. If not present, the stride defaults to 1 along each spatial axis.
+
+ +#### Inputs (2 - 3) + +
+
X : T
+
+
W : T
+
+
B (optional) : T
+
+
+ +#### Outputs + +
+
Y : T
+
+
+ +#### Type Constraints + +
+
T : tensor(float16), tensor(float), tensor(double)
+
Constrain input and output types to float tensors.
+
+ + +### **com.ms.internal.nhwc.DepthToSpace** + +#### Version + +This version of the operator has been available since version 13 of the 'com.ms.internal.nhwc' operator set. + +Other versions of this operator: com.ms.internal.nhwc.DepthToSpace-1, com.ms.internal.nhwc.DepthToSpace-11 + +#### Attributes + +
+
blocksize : int (required)
+
Blocks of [blocksize, blocksize] are moved.
+
mode : string
+
DCR (default) for depth-column-row order re-arrangement. Use CRD for column-row-depth order.
+
+ +#### Inputs + +
+
input : T
+
+
+ +#### Outputs + +
+
output : T
+
+
+ +#### Type Constraints + +
+
T : tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(int8), tensor(int16), tensor(int32), tensor(int64), tensor(bfloat16), tensor(float16), tensor(float), tensor(double), tensor(string), tensor(bool), tensor(complex64), tensor(complex128)
+
Constrain input and output types to all tensor types.
+
+ + +### **com.ms.internal.nhwc.GlobalLpPool** + +#### Version + +This version of the operator has been available since version 2 of the 'com.ms.internal.nhwc' operator set. + +#### Attributes + +
+
p : int
+
p value of the Lp norm used to pool over the input data.
+
+ +#### Inputs + +
+
X : T
+
+
+ +#### Outputs + +
+
Y : T
+
+
+ +#### Type Constraints + +
+
T : tensor(bfloat16), tensor(float16), tensor(float), tensor(double)
+
Constrain input and output types to float tensors.
+
+ + +### **com.ms.internal.nhwc.InstanceNormalization** + +#### Version + +This version of the operator has been available since version 6 of the 'com.ms.internal.nhwc' operator set. + +#### Attributes + +
+
activation : string
+
+
activation_params : list of floats
+
+
epsilon : float
+
The epsilon value to use to avoid division by zero.
+
+ +#### Inputs + +
+
input : T
+
+
scale : T
+
+
B : T
+
+
+ +#### Outputs + +
+
output : T
+
+
+ +#### Type Constraints + +
+
T : tensor(float16), tensor(float), tensor(double)
+
Constrain input and output types to float tensors.
+
+ + +### **com.ms.internal.nhwc.LRN** + +#### Version + +This version of the operator has been available since version 13 of the 'com.ms.internal.nhwc' operator set. + +Other versions of this operator: com.ms.internal.nhwc.LRN-1 + +#### Attributes + +
+
alpha : float
+
Scaling parameter.
+
beta : float
+
The exponent.
+
bias : float
+
+
size : int (required)
+
The number of channels to sum over
+
+ +#### Inputs + +
+
X : T
+
+
+ +#### Outputs + +
+
Y : T
+
+
+ +#### Type Constraints + +
+
T : tensor(float16), tensor(float), tensor(double), tensor(bfloat16)
+
Constrain input and output types to float tensors.
+
+ + +### **com.ms.internal.nhwc.LpPool** + +#### Version + +This version of the operator has been available since version 18 of the 'com.ms.internal.nhwc' operator set. + +Other versions of this operator: com.ms.internal.nhwc.LpPool-11 + +#### Attributes + +
+
auto_pad : string
+
auto_pad must be either NOTSET, SAME_UPPER, SAME_LOWER or VALID. Where default value is NOTSET, which means explicit padding is used. SAME_UPPER or SAME_LOWER mean pad the input so that `output_shape[i] = ceil(input_shape[i] / strides[i])` for each axis `i`. The padding is split between the two sides equally or almost equally (depending on whether it is even or odd). In case the padding is an odd number, the extra padding is added at the end for SAME_UPPER and at the beginning for SAME_LOWER.
+
ceil_mode : int
+
Whether to use ceil or floor (default) to compute the output shape.
+
dilations : list of ints
+
dilation value along each spatial axis of the filter. If not present, the dilation defaults is 1 along each spatial axis.
+
kernel_shape : list of ints (required)
+
The size of the kernel along each axis.
+
p : int
+
p value of the Lp norm used to pool over the input data.
+
pads : list of ints
+
Padding for the beginning and ending along each spatial axis, it can take any value greater than or equal to 0. The value represent the number of pixels added to the beginning and end part of the corresponding axis. `pads` format should be as follow [x1_begin, x2_begin...x1_end, x2_end,...], where xi_begin the number of pixels added at the beginning of axis `i` and xi_end, the number of pixels added at the end of axis `i`. This attribute cannot be used simultaneously with auto_pad attribute. If not present, the padding defaults to 0 along start and end of each spatial axis.
+
strides : list of ints
+
Stride along each spatial axis. If not present, the stride defaults to 1 along each spatial axis.
+
+ +#### Inputs + +
+
X : T
+
+
+ +#### Outputs + +
+
Y : T
+
+
+ +#### Type Constraints + +
+
T : tensor(float16), tensor(float), tensor(double)
+
Constrain input and output types to float tensors.
+
+ + +### **com.ms.internal.nhwc.MaxUnpool** + +#### Version + +This version of the operator has been available since version 11 of the 'com.ms.internal.nhwc' operator set. + +Other versions of this operator: com.ms.internal.nhwc.MaxUnpool-9 + +#### Attributes + +
+
activation : string
+
+
activation_params : list of floats
+
+
kernel_shape : list of ints (required)
+
The size of the kernel along each axis.
+
pads : list of ints
+
Padding for the beginning and ending along each spatial axis, it can take any value greater than or equal to 0. The value represent the number of pixels added to the beginning and end part of the corresponding axis. `pads` format should be as follow [x1_begin, x2_begin...x1_end, x2_end,...], where xi_begin the number of pixels added at the beginning of axis `i` and xi_end, the number of pixels added at the end of axis `i`. This attribute cannot be used simultaneously with auto_pad attribute. If not present, the padding defaults to 0 along start and end of each spatial axis.
+
strides : list of ints
+
Stride along each spatial axis. If not present, the stride defaults to 1 along each spatial axis.
+
+ +#### Inputs (2 - 3) + +
+
X : T1
+
+
I : T2
+
+
output_shape (optional) : T2
+
+
+ +#### Outputs + +
+
output : T1
+
+
+ +#### Type Constraints + +
+
T1 : tensor(float16), tensor(float), tensor(double)
+
Constrain input and output types to float tensors.
+
T2 : tensor(int64)
+
Constrain index tensor to int64
+
+ + +### **com.ms.internal.nhwc.QLinearConvTranspose** + +#### Version + +This version of the operator has been available since version 1 of the 'com.ms.internal.nhwc' operator set. + +#### Attributes + +
+
auto_pad : string
+
auto_pad must be either NOTSET, SAME_UPPER, SAME_LOWER or VALID. Where default value is NOTSET
+
dilations : list of ints
+
dilation value along each spatial axis of the filter. If not present, the dilation defaults to 1 along each spatial axis.
+
group : int
+
number of groups input channels and output channels are divided into.
+
kernel_shape : list of ints
+
The shape of the convolution kernel. If not present, should be inferred from input W.
+
output_padding : list of ints
+
Additional elements added to the side with higher coordinate indices in the output. Each padding value in "output_padding" must be less than the corresponding stride/dilation dimension. By default, this attribute is a zero vector. Note that this attribute doesn't directly affect the computed output values. It only controls the selection of the computed values, so changing this attribute only adds or removes output elements. If "output_shape" is explicitly provided, "output_padding" does not contribute additional size to "output_shape" but participates in the computation of the needed padding amount. This is also called adjs or adjustment in some frameworks.
+
output_shape : list of ints
+
The shape of the output can be explicitly set which will cause pads values to be auto generated. If output_shape is specified pads values are ignored. See doc for details for equations to generate pads
+
pads : list of ints
+
Padding for the beginning and ending along each spatial axis
+
strides : list of ints
+
Stride along each spatial axis. If not present, the stride defaults to 1 along each spatial axis.
+
+ +#### Inputs (8 - 9) + +
+
x : T1
+
+
x_scale : tensor(float)
+
+
x_zero_point : T1
+
+
w : T2
+
+
w_scale : tensor(float)
+
+
w_zero_point : T2
+
+
y_scale : tensor(float)
+
+
y_zero_point : T3
+
+
B (optional) : T4
+
+
+ +#### Outputs + +
+
y : T3
+
+
+ +#### Type Constraints + +
+
T1 : tensor(int8), tensor(uint8)
+
Constrain input type to 8-bit integer tensor.
+
T2 : tensor(int8), tensor(uint8)
+
Constrain filter type to 8-bit integer tensor.
+
T3 : tensor(int8), tensor(uint8)
+
Constrain output type to 8-bit integer tensor.
+
T4 : tensor(int32)
+
Constrain bias type to 32-bit integer tensor.
+
+ + +### **com.ms.internal.nhwc.Resize** + +#### Version + +This version of the operator has been available since version 19 of the 'com.ms.internal.nhwc' operator set. + +Other versions of this operator: com.ms.internal.nhwc.Resize-11, com.ms.internal.nhwc.Resize-13, com.ms.internal.nhwc.Resize-18 + +#### Attributes + +
+
antialias : int
+
If set to 1, "linear" and "cubic" interpolation modes will use an antialiasing filter when downscaling. Antialiasing is achieved by stretching the resampling filter by a factor max(1, 1 / scale), which means that when downsampling, more input pixels contribute to an output pixel.
+
axes : list of ints
+
If provided, it specifies a subset of axes that 'roi', 'scales' and 'sizes' refer to. If not provided, all axes are assumed [0, 1, ..., r-1], where r = rank(data). Non-specified dimensions are interpreted as non-resizable. Negative value means counting dimensions from the back. Accepted range is [-r, r-1], where r = rank(data). Behavior is undefined if an axis is repeated.
+
coordinate_transformation_mode : string
+
+This attribute describes how to transform the coordinate in the resized tensor to the coordinate in the original tensor. + +The coordinate of each dimension is transformed individually. Let's describe a case using axis x as an example. +Denote `x_resized` as the coordinate of axis x in the resized tensor, + `x_original` as the coordinate of axis x in the original tensor, + `length_original` as the length of the original tensor in axis x, + `length_resized` as the length of the resized tensor in axis x, + `scale = length_resized / length_original`, + `output_width` the target length on the axis x which can be a fractional number when it is calculated out of a scale factor, + and `output_width_int` the effective output width as an integer. + +if coordinate_transformation_mode is `"half_pixel"`, +``` +x_original = (x_resized + 0.5) / scale - 0.5 +``` + +if coordinate_transformation_mode is `"half_pixel_symmetric"`, +``` +adjustment = output_width_int / output_width +center = input_width / 2 +offset = center * (1 - adjustment) +x_ori = offset + (x + 0.5) / scale - 0.5 +``` + +if coordinate_transformation_mode is `"pytorch_half_pixel"`, +``` +x_original = length_resized > 1 ? (x_resized + 0.5) / scale - 0.5 : 0 +``` + +if coordinate_transformation_mode is `"align_corners"`, +``` +x_original = x_resized * (length_original - 1) / (length_resized - 1) +``` + +if coordinate_transformation_mode is `"asymmetric"`, +``` +x_original = x_resized / scale +``` + +if coordinate_transformation_mode is `"tf_crop_and_resize"`, +``` +x_original = length_resized > 1 ? start_x * (length_original - 1) + x_resized * (end_x - start_x) * (length_original - 1) / (length_resized - 1) : 0.5 * (start_x + end_x) * (length_original - 1) +``` +.
+
cubic_coeff_a : float
+
The coefficient 'a' used in cubic interpolation. Two common choice are -0.5 (in some cases of TensorFlow) and -0.75 (in PyTorch). Check out Equation (4) in https://ieeexplore.ieee.org/document/1163711 for the details. This attribute is valid only if mode is "cubic".
+
exclude_outside : int
+
If set to 1, the weight of sampling locations outside the tensor will be set to 0 and the weight will be renormalized so that their sum is 1.0. The default value is 0.
+
extrapolation_value : float
+
When coordinate_transformation_mode is "tf_crop_and_resize" and x_original is outside the range [0, length_original - 1], this value is used as the corresponding output value. Default is 0.0f.
+
keep_aspect_ratio_policy : string
+
+This attribute describes how to interpret the `sizes` input with regard to keeping the original aspect ratio of the input, and it is not applicable when +the `scales` input is used. + +Given a set of `sizes`, associated with a subset of `axes` (explicitly provided or default), and assuming `d = axes[i]`, with `i` being the index of the provided `sizes`. + +If `keep_aspect_ratio_policy` is `"stretch"`, the original aspect ratio is disregarded, and the input is resized to the specified size: +`out_size[d] = sizes[i]` + +If `keep_aspect_ratio_policy` is `"not_larger"`, the sizes are adjusted so that no extent of the output is larger than the specified size, while keeping the original aspect ratio: +``` +scale = Min(sizes[i] / in_size[d]) +out_size[d] = round_int(scale * in_size[i]) +``` + +If `keep_aspect_ratio_policy` is `"not_smaller"`, the sizes are adjusted so that no extent of the output is smaller than the specified size, while keeping the original aspect ratio: +``` +scale = Max(sizes[i] / in_size[d]) +out_size[d] = round_int(scale * in_size[i]) +``` + +For non-resizable axes (those not specified in `axes`), the output size will be equal to the input size. + +Note: `round_int` stands for computing the nearest integer value, rounding halfway cases up.
+
mode : string
+
Three interpolation modes: "nearest" (default), "linear" and "cubic". The "linear" mode includes linear interpolation for 1D tensor and N-linear interpolation for N-D tensor (for example, bilinear interpolation for 2D tensor). The "cubic" mode includes cubic interpolation for 1D tensor and N-cubic interpolation for N-D tensor (for example, bicubic interpolation for 2D tensor).
+
nearest_mode : string
+
Four modes: "round_prefer_floor" (default, as known as round half down), "round_prefer_ceil" (as known as round half up), "floor", "ceil". Only used by nearest interpolation. It indicates how to get "nearest" pixel in input tensor from x_original, so this attribute is valid only if "mode" is "nearest".
+
+ +#### Inputs (1 - 4) + +
+
X : T1
+
+
roi (optional) : T2
+
+
scales (optional) : tensor(float)
+
+
sizes (optional) : tensor(int64)
+
+
+ +#### Outputs + +
+
Y : T1
+
+
+ +#### Type Constraints + +
+
T1 : tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(int8), tensor(int16), tensor(int32), tensor(int64), tensor(bfloat16), tensor(float16), tensor(float), tensor(double), tensor(string), tensor(bool), tensor(complex64), tensor(complex128)
+
Constrain input 'X' and output 'Y' to all tensor types.
+
T2 : tensor(float16), tensor(float), tensor(double)
+
Constrain roi type to float or double.
+
+ + +### **com.ms.internal.nhwc.SpaceToDepth** + +#### Version + +This version of the operator has been available since version 13 of the 'com.ms.internal.nhwc' operator set. + +Other versions of this operator: com.ms.internal.nhwc.SpaceToDepth-1 + +#### Attributes + +
+
blocksize : int (required)
+
Blocks of [blocksize, blocksize] are moved.
+
+ +#### Inputs + +
+
input : T
+
+
+ +#### Outputs + +
+
output : T
+
+
+ +#### Type Constraints + +
+
T : tensor(uint8), tensor(uint16), tensor(uint32), tensor(uint64), tensor(int8), tensor(int16), tensor(int32), tensor(int64), tensor(bfloat16), tensor(float16), tensor(float), tensor(double), tensor(string), tensor(bool), tensor(complex64), tensor(complex128)
+
Constrain input and output types to all tensor types.
+
+ + diff --git a/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc b/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc index c5fb9bef26..5c345b8a43 100644 --- a/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc +++ b/onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc @@ -2497,7 +2497,7 @@ TensorrtExecutionProvider::GetCapability(const GraphViewer& graph, const auto& node = graph.GetNode(node_index[i]); const bool is_context_node = node && !node->OpType().empty() && node->OpType() == "EPContext"; if (is_context_node) { - SubGraph_t supported_node_vector = {std::vector{i}, true}; + SubGraph_t supported_node_vector (std::vector{i}, true); std::unique_ptr sub_graph = GetSubGraph(supported_node_vector, graph, model_hash, subgraph_idx++); result.push_back(ComputeCapability::Create(std::move(sub_graph))); }