This PR is to add support of using env variable to set provider option
cudnn_conv_algo_search so that user can choose better conv algo search
method to run model. This is a quick fix to unblock the test of MoE
model. Will have another PR to design and implement the ORTModule config
so that we can config ORTModule using Python script or config file
instead of env variable.
Model [huggingface's diffusers
library](https://github.com/huggingface/diffusers) has
torch.nn.GroupNorm which will be exported to sub-graph containing ONNX's
InstanceNormalization, which is lack of gradient. The implementation of
ORT's InstanceNormalization will call cuDNN's BatchNorm for part of
computation, which is not efficient compared to PyTorch's
implementation. This PR is to use ATen fallback to support this torch
module, including its forward and backward.
### Description
float and half initializers with same value are merged into same
initializer. This is a bug due to when we create pattern key, data type
is always be -1 (which is a naive mistake when doing code refactoring
previously), plus float and half are stored as float in constant store
for easier data comparison.
Added test coverage.
` [ONNXRuntimeError] : 1 : FAIL : Type Error: Type parameter (T) of
Optype (Mul) bound to different types (tensor(float) and tensor(float16)
in node
`
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
<!-- Describe your changes. -->
Unit test with ROCm5.3 slower than ROCm5.2.3. Revert to ROCm5.2.3.
We will update to ROCm5.3 when the issue resloved by AMD.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
<!-- Describe your changes. -->
Add ability to upgrade an ORT format model when loaded in a full build
by inserting the kernel constraint info and ignoring the kernel hashes.
This also allows upgrading the model to the latest format by saving the
model after loading.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
Provide official path to upgrading an ORT format model directly (vs.
reconverting).
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
### Description
supplement of #13248
Add PR trigger
https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=azure-devops&tabs=yaml#pr-triggers
fix: master -> main
Testted with #13289#13292
NB:
the real pipeline is always triggered if the workflow yaml changed even
it's added in the path filter.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Make sure the real pipeline not run in the backend.
### Description
Check for null input
### Motivation and Context
This has been reported at least twice (once by the Windows team and once
by Speech team). Currently we just segfault.
Current graph builder for ORTModule will apply the training's graph
optimizations for both training and eval mode. Take BatchNorm as
example, one of training's graph optimizations will replace
BatchNormalization Op to BatchNormInternal which is for training only.
This PR is to fix this, for eval mode, we will not apply the training's
graph optimizations. The inference's graph optimizations will be applied
when InferenceSession initialization.
### Description
Correct the file name in the comments of the generated yaml.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
<!-- Describe your changes. -->
1. Remove ROCm5.1.1 and ROCm5.2 from ROCm python package pipeline
2. Add ROCm5.3 to ROCm python package pipeline
pipeline:
https://aiinfra.visualstudio.com/Lotus/_build/results?buildId=237172&view=results
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
Implemented gradient of cos as per the function below.
### Motivation and Context
Cos gradient required for [huggingface's diffusers
library](https://github.com/huggingface/diffusers)
### Testing
built ORT from source: `./build.sh --config RelWithDebInfo
--enable_training --use_cuda --cuda_home /usr/local/cuda --cudnn_home
/usr/local/cuda --build_wheel --parallel --skip_tests`
tested CosGrad implementation: `cd build/Linux/RelWithDebInfo/ &&
./onnxruntime_test_all --gtest_filter=GradientCheckerTest.CosGrad`
Co-authored-by: Prathik Rao <prathikrao@microsoft.com>
### Description
Implemented gradient of sin as a function op.
### Motivation and Context
Sin gradient currently implemented as cpu op which could hurt
performance.
### Testing
built ORT from source: `./build.sh --config RelWithDebInfo
--enable_training --use_cuda --cuda_home /usr/local/cuda --cudnn_home
/usr/local/cuda --build_wheel --parallel --skip_tests`
tested SinGrad implementation: `cd build/Linux/RelWithDebInfo/ &&
./onnxruntime_test_all --gtest_filter=GradientCheckerTest.SinGrad`
Co-authored-by: Prathik Rao <prathikrao@microsoft.com>
Co-authored-by: Baiju Meswani <bmeswani@microsoft.com>
### Description
To construct test name, replace whitespace to underscore and remove
parentheses
### Motivation and Context
gtest name only accepts '_' and alphanumeric
**Description**: Add qkv_hidden_size support in CUDA Attention Layer
implementation.
Changes include:
- Modify UT to test GPU and CPU implementation
- Add overload for CUDA kernel `AddBiasTransposeQKV` to support scenario
where V_HIDDEN_SIZE != QK_HIDDEN_SIZE
- Update variable names from `head_size` to `qkv_head_sizes[0]` or
`qkv_head_sizes[2]`
- Modify function definitions to allow communication of
`qkv_hidden_sizes` or `qkv_head_sizes`
Note that this feature is not supported in Rocm EP or quantized
attention right now.
**Motivation and Context**
- Why is this change required? What problem does it solve? The current
CUDA implementation of attention layer doesn't support the parameter
qkv_hidden_size added in the CPU implementation in PR
[8039](https://github.com/microsoft/onnxruntime/pull/8039)
- If it fixes an open issue, please link to the issue here.
Co-authored-by: Peter Mcaughan <petermca@microsoft.com>
This PR has two fixes:
- https://github.com/pytorch/pytorch/pull/85636 change the behavior of
register_custom_op_symbolic to only register the symbolic function at a
single version. For ORTModule we need to pass the op_set version when
calling it.
- Since torch_1.13 the signature of einsum is changed to have a new
argument, need to change our custom op symbolic registry code
accordingly.
Without the fixes, ORTModule will not work with the nightly torch, and
the new torch version will be released.
clang-tidy says "Do not implicitly decay an array into a pointer; consider using gsl::array_view or an explicit cast instead"
It is a false positive scattering around all our codebase when using
helper macros. It is becuase for function with 4 char name, say `main`,
the type of __FUNCTION__ and __PRETTY_FUNCTION__ is `char [5]`.
**Description**:
1. Share scalar constant for same data type, value and shape.
2. Fix the order of Graph resolve context clear and
CleanUnusedInitializersAndNodeArgs().
**Share initializer for those who hold same value in same type and
shape, currently only handle scalar value or 1-D single value array.**
The transformation itself did not bring much impact on memory/perf,
instead is helpful to simplify the graph, making it easier for common
subexpression eliminations (CSE). Imagine graphs like this:
Add is NOT shared as inputs of Clip after CSE transformation because,
all Add's second constant input are different NodeArg*, so if we change
all constant initializer share the same NodeArg*, then only one Add will
be preserved after CSE transformation. There are few other similar cases
in one of 1P deberta models.
E2E measurement on 1P DEBERTA model, we see an increase from
SamplesPerSec=562.041593991271 to 568.0106130440271, 1.07% gains.
**Fix the order of Graph resolve context clear and
CleanUnusedInitializersAndNodeArgs().**
Graph resolve context will be cleared every time by end of
Graph.Resolve(), one of the thing to be cleared is the
"inputs_and_initializers" who hold string_view of all initializers.
While CleanUnusedInitializersAndNodeArgs removed some initializers, so
some strings that is referenced by string_view in
"inputs_and_initializers" remain to be there BUT in an invalid state.
**Motivation and Context**
- Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here.
### Description
Re-architect DML EP to allow ORT L2/L3 transformers. This change
includes:
- During ORT graph partitioning, DML EP will only set the
dmlExecutionProvider to all eligible nodes.
- Moved DML specific operator transformer as L2 transformer
- Introduced a new DMLGraphFusionTransformer, applicable only for DML
EP, which is responsible to
- partition the graph
- fuse each partition into a IDMLCompiledOperator
- register the kernel for each partition
### Motivation and Context
- Why is this change required? What problem does it solve?
It enables ORT L2/L3 transformers for DML EP, which will increase the
perf of Transformer-based models.
- If it fixes an open issue, please link to the issue here. N/A
Co-authored-by: Sumit Agarwal <sumitagarwal@microsoft.com>
### Description
<!-- Describe your changes. -->
fix some typo in docs
### Motivation and Context
singed vs signed
succeding vs succeeding
fileter vs filter
kernal vs kernel
libary vs library
Fix warnings and enable dev mode for ROCm CI:
* Fix ROCm headers complaining "This file is deprecated. Use the header file from ..."
* Disable warning signed and unsigned compare for kernel explorer
* Fix unused and nondiscard warnings
* Enable dev mode for ROCm CI
* Walkaround error "unknown warning option '-Wno-nonnull-compare'" in kernel explorer by using '-Wno-unknown-warning-option' to ignore the unknown option
* Fix error "unused parameter 'mask'"
* Fix warning "instantiation of variable 'onnxruntime::rocm::Consts<float>::One' required here, but no definition is available", etc. Fixed by using C++17's inline (implied by constexpr) static initialization.
* Remove unused variable
* Add the missing `override` specifier
### Description
Deprecate CustomOpApi and refactor dependencies for exception safety and
eliminate memory leaks.
Refactor API classes for clear ownership and semantics.
Introduce `InitProviderOrtApi()`
### Motivation and Context
Make public API better and safer.
Special note about `Ort::Unowned`. The class suffers from the following
problems:
1. It is not able to hold const pointers to the underlying C objects.
This forces users to `const_cast` and circumvent constness of the
returned object. The user is now able to call mutating interfaces on the
object which violates invariants and may be a thread-safety issue. It
also enables to take ownership of the pointer and destroy it
unintentionally (see examples below).
2. The objects that are unowned cannot be copied and that makes coding
inconvenient and at times unsafe.
3. It directly inherits from the type it `unowns`.
All of the above creates great conditions for inadvertent unowned object
mutations and destructions. Consider the following examples of object
slicing, one of them is from a real customer issue and the other one I
accidentally coded myself (and I am supposed to know how this works).
None of the below can be solved by aftermarket patches and can be hard
to diagnose.
#### Example 1 slicing of argument
```cpp
void SlicingOnArgument(Ort::Value& value) {
// This will take possession of the input and if the argument
// is Ort::Unowned<Ort::Value> it would again double free the ptr
// regardless if it was const or not since we cast it away.
Ort::Value output_values[] = {std::move(value)};
}
void main() {
const OrtValue* ptr = nullptr; // some value does not matter
Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)};
// onowned is destroyed when the call returns.
SlicingOnArgument(unowned);
}
```
#### Example 2 slicing of return value
```cpp
// The return will be sliced to Ort::Value that would own and relase (double free the ptr)
Ort::Value SlicingOnReturn() {
const OrtValue* ptr = nullptr; // some value does not matter
Ort::Unowned<Ort::Value> unowned{const_cast<OrtValue*>(ptr)};
return unowned;
}
```
Add BART into transformer support, specificalyy for
`BartForConditionalGeneration`
**Motivation and Context**
- fixes#11210
Currently, the custom op beam search is not working in nightly, this PR
should be run with a [custom
commit](10f3d46d92)
### Description
<!-- Describe your changes. -->
As title.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Uint8 type might be required for some model used in sample application.
To match supported data types for onnxruntime-react-native for Android.
Co-authored-by: rachguo <rachguo@rachguos-Mac-mini.local>
Co-authored-by: rachguo <rachguo@rachguos-Mini.attlocal.net>
* Change block dimension type to Int from Ints.
* In response to feedback that the block dimension corresponds to the
reduction dimension of the consuming matrix multiplication. There is
always only 1 reduction dimension.
### Description
<!-- Describe your changes. -->
Add special case handling for exclusive + reverse where axis has dim
value of 1.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
#13165
### Description
binraries ==> binaries
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Update for ROCm CI before reland tunable GEMM #12853. This PR also update
composable kernel to use CMakes's HIP language support so that we can
mix C/C++ compiler with HIP compiler instead of locking to hip-clang
### Description
Training C# bindings (ReleaseTrainingSession and ReleaseCheckpointState)
broke after an API order change in Training C API. This PR fixes this
issue.
### Motivation and Context
Bug Fix for Training C# bindings
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
