### Description
This PR fixes a bug when the ONNX checker is called while exporting
Whisper for FP16 CUDA with optional flags.
### Motivation and Context
Sometimes, the ONNX checker raises an error depending on the optional
flags passed. By wrapping the ONNX checker in a try-except, the
conversion can continue even if the checker fails.
### Description
Add additonal gfx targets for AMD GPU support
### Motivation and Context
Required to integrate mainline onnxruntime support for AMD GPUs
---------
Co-authored-by: Stefan Sokolovic <stsokolo@amd.com>
Co-authored-by: Jeff Daily <jeff.daily@amd.com>
[TensorRT-Model-Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer)
have a implementation for INT4 AWQ. Adding the support in onnxruntime
tools to quantized the models with TensorRT-Model-Optimizer
### Description
* Add digital signature to dll files in jar files.
* Jar file names: onnxruntime-{version}.jar,
onnxruntime_gpu-{version}.jar
### Motivation and Context
#19204
### Description
<!-- Describe your changes. -->
flatbuffers always write data in LE and it is automatically traslated
to/from BE as needed,
but only if we use proper accessors. This would work for shape.
However, we store parameters as bytes, so we need to swap bytes as
needed for BE.
### Motivation and Context
Address https://github.com/microsoft/onnxruntime/issues/22364
### Description
Aallows alpha, beta and rc version releases to Maven for Android
artifacts.
### Motivation and Context
Helpful to release rc versions or test artifacts to Maven for testing.
For example, a new QNN android package is being released and it will be
nice to test the RC version for dependencies before release
## Future Work
Allow RC version for all Maven artifacts.
### Description
<!-- Describe your changes. -->
Pick up onnx/onnx#6010 to support EinSum shape inference
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
This change allows EinSum operator's output shape to be inferenced so
that it can run on accelerators.
### Description
Pre built QNN Android package
### Future Work
1. Setting up CI with Browserstack- onnxruntime_tests and Android test
2. ESRP Release to Maven
### Description
In TensorRT 10.5, the APIs `platformHasFastFp16` and
`platformHasFastInt8` have been deprecated.
Ignore these deprecation warnings.
Signed-off-by: Kevin Chen <kevinch@nvidia.com>
### Description
Update segment anything 2 benchmark script:
(1) Fix cuda graph in benchmark. Make sure --use_cuda_graph takes effect
and random_inputs() generates according to the dtype of the model.
(2) Add a parameter to enable profiling.
(3) Use latest cuda 12.6.2 and cudnn 9.5.
(4) Update README.md.
### Motivation and Context
Previous, --use_cuda_graph does not take effect. This fixes the
benchmark.
### Description
Add SetEpDynamicOptions and Remove workload_type from run/session
options.
### Motivation and Context
Added SetEpDynamicOptions as a dynamic way of changing EP settings even
in the middle of a Run
Using workload_type run/session options to set Efficient/Default mode
for workloads does not cover all the scenarios and can lead to priority
inversions. Working on a new API to support setting Efficient/Default
mode for workloads.
---------
Co-authored-by: Luis E. Pena <luispena@microsoft.com>
### Description
Resolve#21976 .
ABSL generally does not have forward/backward compatibility. Our code is
only compatible with one fixed LTS version. So it's important to fix the
version number there when using find_package to detect an installed
version.
### Description
It runs after "Python-CUDA-Packaging-Pipeline" that runs on a CPU
machine that skipped all tests.
This testing pipeline is for doing the tests.
This fixes a bug found by libfuzzer:
LayerNormalization third input (beta) is optional. The following code
has potential out of bound access if the input is not available:
```
NodeArg* beta = layer_norm_node.MutableInputDefs()[2];
```
This adds a check to ensure the third input exists before fusion.
[AB#49036](https://aiinfra.visualstudio.com/6a833879-cd9b-44a4-a9de-adc2d818f13c/_workitems/edit/49036)
### Description
* Add a few arguments CUDA_VERSION, CUDNN_VERSION, OS, GIT_COMMIT,
GIT_BRANCH and ONNXRUNTIME_VERSION to the Dockerfile.cuda to allow for
more flexibility in the build process.
* Update README.md to include the new arguments and their usage.
* Output labels to image so that it is easy to inspect the image.
Available CUDA versions for ubuntu 24.04 can be found
[here](https://hub.docker.com/r/nvidia/cuda/tags), and available CUDNN
versions can be found
[here](https://pypi.org/project/nvidia-cudnn-cu12/#history). Example
command line to build docker image:
```
docker build -t onnxruntime-cuda --build-arg CUDA_VERSION=12.6.1 \
--build-arg CUDNN_VERSION=9.5.0.50 \
--build-arg GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD) \
--build-arg GIT_COMMIT=$(git rev-parse HEAD) \
--build-arg ONNXRUNTIME_VERSION=$(cat ../VERSION_NUMBER) \
-f Dockerfile.cuda ..
```
Example labels from `docker inspect onnxruntime-cuda`:
```
"Labels": {
"CUDA_VERSION": "12.6.1",
"CUDNN_VERSION": "9.5.0.50",
"maintainer": "Changming Sun <chasun@microsoft.com>",
"onnxruntime_git_branch": "main",
"onnxruntime_git_commit": "bc84958dcef5c6017ae58085f55b669efd74f4a5",
"onnxruntime_version": "1.20.0",
"org.opencontainers.image.ref.name": "ubuntu",
"org.opencontainers.image.version": "24.04"
}
```
### Motivation and Context
https://github.com/microsoft/onnxruntime/pull/22339 has hard-coded the
cuda and cudnn versions. User might want to choose specified cuda and
cudnn version during building docker image.
Fix the QNN nuget package issue
### Description
Inside the package, folder name \runtimes\win-arm64\ was changed to \runtimes\win-ARM64\, which breaks lib copy settings in Microsoft.ML.OnnxRuntime.QNN.props.
### Motivation and Context
Fix issue: https://github.com/microsoft/onnxruntime/issues/21692
### Description
Update the commit from 59600894a2c1c18290944b83e989bfe618975230 to
1887322ed36d522409a6b805d4e7942cf76a8e40
### Motivation and Context
The new one has python 3.13.
AB#50959
This reverts commit 4e15b229a0.
Reason: We are seeing an increase in the number of deadlocks after this
PR. We have a release coming up next week and do not have enough time to
investigate the root cause, hence reverting this PR temporarily.
Moreover, this is causing an increase int he binary size.
### Description
We are seeing an [increase in the number of
deadlocks](https://github.com/microsoft/onnxruntime/pull/22315#issuecomment-2394821893)
after this PR. We have a release coming up next week and do not have
enough time to investigate the root cause, hence reverting this PR
temporarily.
### Motivation and Context
See above.
### Description
This change introduces the WebGPU EP into ONNX Runtime.
To make the PR as simple as possible, this PR excluded the following:
- C API changes for WebGPU EP
- actual implementation of WebGPU EP. Currently in this PR, WebGPU is a
stub implementation that does not register any kernel.
- Python IO Binding update
- Node.js IO Binding update
This PR now contains only 43 file changes (while the working branch
contains 130+) and hopefully this makes it easier to review.
There is going to be separated PRs for each mentioned above.
Current working branch: #21904
### Description
Serve as example to build and run onnxruntime-gpu with latest software
stack.
To build docker image:
```
git clone https://github.com/microsoft/onnxruntime
cd onnxruntime/dockerfiles
docker build -t onnxruntime-cuda -f Dockerfile.cuda ..
```
To launch the docker image built from previous step (and mount the code
directory to run a unit test below):
```
cd ..
docker run --rm -it --gpus all -v $PWD:/code onnxruntime-cuda /bin/bash
```
Then run the following in docker image to verify that the cuda provider
is good:
```
python /code/onnxruntime/test/python/onnxruntime_test_python_cudagraph.py
```
### Motivation and Context
https://github.com/microsoft/onnxruntime/issues/22335
### Description
In GQA there was a memory issue which was best described by @edgchen1
[here](https://github.com/microsoft/onnxruntime/issues/22252#issuecomment-2384559255)
> here's the problematic code:
>
>
d9de054eb5/onnxruntime/contrib_ops/cpu/bert/group_query_attention.cc (L149-L157)
>
> annotated:
>
> ```c++
> if (packed_qkv) {
> // Q is an OrtValue declared in the enclosing scope.
> OrtValue RotaryQKV;
> Tensor::InitOrtValue(element_type, TensorShape({batch_size, num_heads_
+ 2 * kv_num_heads_, sequence_length, head_size}), allocator,
RotaryQKV);
> // Save pointer to Q's data in q_input.
> q_input = Q.Get<Tensor>().Data<T>();
> k_input = q_input + num_heads_ * sequence_length * head_size;
> q_rotary = RotaryQKV.GetMutable<Tensor>()->MutableData<T>();
> k_rotary = q_rotary + num_heads_ * sequence_length * head_size;
> // Overwrite Q with RotaryQKV (OrtValues contain shared_ptr to
contained value).
> // Now, q_input is pointing to freed memory.
> Q = RotaryQKV;
> }
> ```
>
> later on, when we use `q_input`, there is a read access violation.
>
>
d9de054eb5/onnxruntime/contrib_ops/cpu/bert/group_query_attention.cc (L170-L172)
>
> this problem showed up when CPU allocator sharing between sessions was
enabled. in that case, the CPU allocator's arena was disabled. I suspect
that the default usage of the arena hid this issue.
>
> though I debugged into the first branch, this appears to be a problem
in both branches:
>
>
d9de054eb5/onnxruntime/contrib_ops/cpu/bert/group_query_attention.cc (L149-L168)
### Motivation and Context
Fixes a crucial bug. The issue was found here
https://github.com/microsoft/onnxruntime/issues/22252
### Description
Add ROCm EP option to benchmark.py script when using int8 quantization.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Without this change benchmarks with int8 quantization cannot be run with
ROCm execution provider.
### Description
<!-- Describe your changes. -->
Specify type to fix warning
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
Increanse TensorRT tolerance from default 1e-5 to 1e-3 after TRT 10.4
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
Add interface to get config_options from onnxruntime.
### Motivation and Context
to support config session_options after EP Append, So need get
configurations on ep end.
### Description
(1) Support onnx data types in python APIs:
* IOBinding.bind_input
* IOBinding.bind_output
* ortvalue_from_shape_and_type
(2) Add unit tests, which serves an example of running BFloat16 or
Float8 models in Python.
Other minor changes:
(3) replace deprecated NP_TYPE_TO_TENSOR_TYPE by helper API.
(4) Rename ortvalue_from_numpy_with_onnxtype to
ortvalue_from_numpy_with_onnx_type.
The integer of onnx element type can be found in
(https://onnx.ai/onnx/api/mapping.html). Note that FLOAT4E2M1 is not
supported yet.
### Motivation and Context
Current python API does not support Bfloat16 and float8 (FLOAT8E4M3FN,
FLOAT8E4M3FNUZ, FLOAT8E5M2, FLOAT8E5M2FNUZ) types, and other new data
types like INT4, UInt4 etc.
This removes the limitation.
https://github.com/microsoft/onnxruntime/issues/13001https://github.com/microsoft/onnxruntime/issues/20481https://github.com/microsoft/onnxruntime/issues/20578
### Description
With TensorRT 10.4 update, the name of TensorRT windows package changed
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
