Add Gelu/QuickGelu/GeluGrad/QuickGeluGrad support to Triton Codegen so
that it can be fused with some other connected supported Ops. For
example, in llama2, it can be fused with Mul so we will have extra 1-2%
perf gain.
### Description
<!-- Describe your changes. -->
Add ability for transpose optimizer to look past a DQ node if it has a
constant initializer as input. This allows UnsqueezeInput/TransposeInput
to modify the initializer in-place in the same way it would for a
non-QDQ format model.
Shared initializers are also handled, and any additional
Squeeze/Transpose added to the other usages of the initializer should
cancel out when we push the same Transpose though them.
The in-place modification means we don't need to run QDQ fixup and
constant folding after layout transformation. This means we do not need
to enable those optimizers in a minimal build to get an optimal model
post-layout transformation.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Ensure layout transformation produces optimal model in full and minimal
builds.
---------
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
### Description
<!-- Describe your changes. -->
* Allow either an allocator or a MemBuffer to be used when creating an
OrtValue from an TensorProto
* `Tensor<std::string>` requires an allocator to allocate/free the
string values
* Forcing the buffer to be allocated outside of the Tensor doesn't seem
to provide any benefit in this usage as the Tensor class disables copy
and assignment (so we wouldn't create 2 copies of the buffer via the
Tensor class that externally managing the would buffer avoid)
* New approach means we don't need to manage the buffers in the
optimizer Info class as the Tensor dtor will do that
* Update naming - MLValue was replaced by OrtValue a long time ago
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
#17392
### Description
<!-- Describe your changes. -->
Fix for this issue which raise the error of FileNotAccessd in windows
when the context of TemporaryDirectory finished.
https://github.com/microsoft/onnxruntime/issues/17627
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
https://github.com/microsoft/onnxruntime/issues/17627
### Description
this is for ORT 1.17.0 - make ORT to use ONNX release 1.15.0 branch. Eventually will update to the release tag once ONNX 1.15.0 is released
### Motivation and Context
Prepare for ORT 1.17.0 release. People can start work on new and updated ONNX ops in ORT.
---------
Signed-off-by: Liqun Fu <liqfu@microsoft.com>
### Description
Updated a couple of old links in the technical documentation that where
pointing to files present prior to the migration to
https://onnxruntime.ai/docs.
### Description
<!-- Describe your changes. -->
For now, WebNN Softmax only support 2D (or implicitly coerce to 2D)
inputs and the last axis.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Fallback some cases to pass the CI.
### Description
Adds prepacked weights container to model subgraphs.
### Motivation and Context
Allows for initializer sharing when the initializers are located in
subgraphs. I encountered this bug when attempting to share weights
between T5 BeamSearch models where the shareable initializers are
located in the encoder and decoder subgraphs and it failed to reduce
memory usage.
### Description
<!-- Describe your changes. -->
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
This patch adds a closing curly bracket at the end of `settings.json`.
### Motivation and Context
`settings.json` is just not closed. It was accidentally removed at
4e6ea730d6
Stop throwing the exception when the provider list is empty but there
are multiple available EPs.
Other language bindings throw no exception at all, this change will
align them up.
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
1. Upgrade nodejs from 16.x to 18.x for Windows pipelines
2. Avoid using Azure DevOps "NodeTool" on Linux. The tool installs
nodejs from internet or local disk cache. But we already moved all Linux
tests to docker. So we do not need the installer anymore.
3. Remove some other unused code.
### Description
update Chromium browser launch command line flags
Canary already using dxc so no need to specify
'--enable-dawn-features=use_dxc' for canary.
### Description
allow DataTransfer to deal with zero sized input.
This is a standalone fix for zero-sized tensor handling for JSEP
DataTransfer. There are other components in JSEP not supporting
zero-sized tensors need to be fixed.
### Description
allow JsCustomAllocator to deal with zero sized input.
This is a standalone fix for zero-sized tensor handling for
JsCustomAllocator. There are other components in JSEP not supporting
zero-sized tensors need to be fixed.
### Description
One quantization case was not covered by the current list of unit tests.
This PR adds a unit test to cover that case with the fix. It fixes the
issue #17619.
### Motivation and Context
### Description
ort-web build step - webpack consumes the amount of memory on the edge
of Node.js(V8)'s default max-old-space-size, so increase the default
memory size to 5GB to avoid this issue.
### Description
QNN SDK version 2.14.1 fixed several issues with the QNN Resize
operator. This PR integrates the fixes and simplifies the
implementation.
### Motivation and Context
Improve Resize operator and test coverage.
Fix ARMv7 build error on Linux.
### Description
`cpuinfo_*` functions are only available if `CPUINFO_SUPPORTED` set and
therefore `"cpuinfo.h"` included.
Fixed with extended conditional code.
### Motivation and Context
Compilation with ARMv7 on Linux system fails.
### Description
This PR optimizes the gather op, which is improved ~6ms in segment
anything model in ADL.
The problem in original algorithm is that it includes a for loop to
calculate a block size of data. However, the block size may be very
large, like `65536`. In GPU shader, we should try to avoid large loop in
shader and try to use more threads to do it parallelly.
Before:
```
[profiling] kernel "41771992|[Gather] 41771992" input[0]: [4,65536] | float32, input[1]: [1] | int64, output[0]: [1,65536] | float32, execution time: 6886207 ns
```
After:
```
[profiling] kernel "41771992|[Gather] 41771992" input[0]: [4,65536] | float32, input[1]: [1] | int64, output[0]: [1,65536] | float32, execution time: 11719 ns
### Description
1. For binary ops, the components is always 4. So the dispatchGroup
should be : `{x: Math.ceil(outputSize / 64 /* workgroup size */ / 4 /*
component size */)}` instead of `{x: Math.ceil(outputSize / 64 /*
workgroup size */ / (vectorize ? 4 : 1) /* vec size */)}`.
2. If any of a or b only has one element, we still can use the vectorize
path since the same value will be broadcasted.
### Description
Add float16 and bfloat16 data type support for VitisAI ep
### Motivation and Context
The VitisAI ep has added the bfloat datatype support. So we would like
to register the datatype from onnxruntime side to enable them.
---------
Signed-off-by: Yiming Hu <yiming.hu@amd.com>
### Model post process for zero stage3 training
This is the last change to make single GPU/Multiple GPUs run pass.
Design details:
https://microsoft.sharepoint.com/:p:/t/ONNX2/EfNfJ43necpIoPI6x5M2zvYBVbfjoPQmG4Boc_F7-tHm1w?e=ekQwA6&nav=eyJzSWQiOjMxNiwiY0lkIjoxMDE1Nzg3NDZ9
`PyTorch` runs with ZeROOffloadSubscriber:
```
model = prepare_model(...)
from onnxruntime.training.utils.hooks import configure_ort_compatible_zero_stage3
configure_ort_compatible_zero_stage3()
```
`ORTModule` runs with ZeROOffloadSubscriber:
```
os.environ['ORTMODULE_ENABLE_ZERO_STAGE3'] = '1'
from onnxruntime.training.ortmodule import ORTModule
model = ORTModule(self.model)
```
It will be fairly easy to debug convergence issue if both ORT and
PyTorch can run the same offload path.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
<!-- Describe your changes. -->
Layer norm fusion changes required for deepspeed stage 3, also includes
test case.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
It helps fusing layer norm for Deepspeed Stage 3. Added a test case
scenario which ensures that the fusion is working properly for the
scenario.
This adds an additional check before enabling MlasGemmU8S8DispatchAmx
for GEMM operations. After checking the CPUID for AMX-TILE and AMX-INT8,
an additional check is added that checks value of the XCR0 register.
The value in the OXR0 register is set by the OS and indicates support
for various CPU features. In this case the bits indicating XTILECFG and
XTILEDATA support are checked.
### Description
This adds an additional check before enabling MlasGemmU8S8DispatchAmx
for GEMM operations. After checking the CPUID for AMX-TILE and AMX-INT8,
an additional check is added that checks value of the XCR0 register.
The value in the OXR0 register is set by the OS and indicates support
for various CPU features. In this case the bits indicating XTILECFG and
XTILEDATA support are checked.
### Motivation and Context
Fix for crash reported directly by customer. When running older Windows
server OS on newer Gen4 Xeon processors.
Signed-off-by: Nash <george.nash@intel.com>
### Description
1. Remove 'dnf update' from docker build scripts, because it upgrades TRT
packages from CUDA 11.x to CUDA 12.x.
To reproduce it, you can run the following commands in a CentOS CUDA
11.x docker image such as nvidia/cuda:11.8.0-cudnn8-devel-ubi8.
```
export v=8.6.1.6-1.cuda11.8
dnf install -y libnvinfer8-${v} libnvparsers8-${v} libnvonnxparsers8-${v} libnvinfer-plugin8-${v} libnvinfer-vc-plugin8-${v} libnvinfer-devel-${v} libnvparsers-devel-${v} libnvonnxparsers-devel-${v} libnvinfer-plugin-devel-${v} libnvinfer-vc-plugin-devel-${v} libnvinfer-headers-devel-${v} libnvinfer-headers-plugin-devel-${v}
dnf update -y
```
The last command will generate the following outputs:
```
========================================================================================================================
Package Architecture Version Repository Size
========================================================================================================================
Upgrading:
libnvinfer-devel x86_64 8.6.1.6-1.cuda12.0 cuda 542 M
libnvinfer-headers-devel x86_64 8.6.1.6-1.cuda12.0 cuda 118 k
libnvinfer-headers-plugin-devel x86_64 8.6.1.6-1.cuda12.0 cuda 14 k
libnvinfer-plugin-devel x86_64 8.6.1.6-1.cuda12.0 cuda 13 M
libnvinfer-plugin8 x86_64 8.6.1.6-1.cuda12.0 cuda 13 M
libnvinfer-vc-plugin-devel x86_64 8.6.1.6-1.cuda12.0 cuda 107 k
libnvinfer-vc-plugin8 x86_64 8.6.1.6-1.cuda12.0 cuda 251 k
libnvinfer8 x86_64 8.6.1.6-1.cuda12.0 cuda 543 M
libnvonnxparsers-devel x86_64 8.6.1.6-1.cuda12.0 cuda 467 k
libnvonnxparsers8 x86_64 8.6.1.6-1.cuda12.0 cuda 757 k
libnvparsers-devel x86_64 8.6.1.6-1.cuda12.0 cuda 2.0 M
libnvparsers8 x86_64 8.6.1.6-1.cuda12.0 cuda 854 k
Installing dependencies:
cuda-toolkit-12-0-config-common noarch 12.0.146-1 cuda 7.7 k
cuda-toolkit-12-config-common noarch 12.2.140-1 cuda 7.9 k
libcublas-12-0 x86_64 12.0.2.224-1 cuda 361 M
libcublas-devel-12-0 x86_64 12.0.2.224-1 cuda 397 M
Transaction Summary
========================================================================================================================
```
As you can see from the output, they are CUDA 12 packages.
The problem can also be solved by lock the packages' versions by using
"dnf versionlock" command right after installing the CUDA/TRT packages.
However, going forward, to get the better reproducibility, I suggest
manually fix dnf package versions in the installation scripts like we do
for TRT now.
```bash
v="8.6.1.6-1.cuda11.8" &&\
yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo &&\
yum -y install libnvinfer8-${v} libnvparsers8-${v} libnvonnxparsers8-${v} libnvinfer-plugin8-${v} libnvinfer-vc-plugin8-${v}\
libnvinfer-devel-${v} libnvparsers-devel-${v} libnvonnxparsers-devel-${v} libnvinfer-plugin-devel-${v} libnvinfer-vc-plugin-devel-${v} libnvinfer-headers-devel-${v} libnvinfer-headers-plugin-devel-${v}
```
When we have a need to upgrade a package due to security alert or some
other reasons, we manually change the version string instead of relying
on "dnf update". Though this approach increases efforts, it can make our
pipeines more stable.
2. Move python test to docker
### Motivation and Context
Right now the nightly gpu package mixes using CUDA 11.x and CUDA 12.x
and the result package is totally not usable(crashes every time)
### Description
Include onnxruntime_float16.h in the package.
### Motivation and Context
This was missed in the recently released 1.16 pkgs (except Nuget).
manylinux build is used for nightly packaging generation and it's hard
to capture issue in time when related files change. This PR add
manylinux build in CI.
### Description
Adds more operator unit tests (all op types should now have at least 1
unit test):
- [x] Reshape
- [x] Flatten
- [x] Squeeze
- [x] Unsqueeze
- [x] Gemm
- [x] Clip
- Enable QDQ Clip on HTP backend (when not optimized away by L1
ClipQuantFusion optimizer)
- Add support for 16-bit QDQ Clip to ClipQuantFusion optimizer
- [x] Split
- [x] Topk
- Enable QDQ TopK on HTP backend
- [x] Tile
- Enable QDQ Tile on HTP backend
### Motivation and Context
Increase QNN operator support and test coverage.
The extensions submodule was removed in [this
PR](https://github.com/microsoft/onnxruntime/pull/17097) but not deleted
from the list of git modules. This causes breaks in code ingesting ORT
that references the git modules for an accurate list of submodules.
This change removes the extensions from the list of git modules to
resolve this issue.
