### Description
Our nightly CPU python package's name is "ort-nightly" instead of
"onnxruntime". It was because of some historical reasons. Tensorflow was
like that.
Now we would prefer to make them the same.
Do this change for all nightly python packages, including CPU,
GPU(CUDA), and maybe others.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
* Add in missing operators for llama run
* Add simplified layer norm ops
### Description
<!-- Describe your changes. -->
Adding additional supported operators into MIGraphX EP that are
supported in MIGraphX
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Allows for more models to be run through MIGraphX EP
### Description
Today, stable diffusion stage failed due to there's a upgrade in timm.
controlnet_aux depends on it.
And its latest version limit the timm version less than 0.6.7.
So upgrading controlnet_aux can solve it.
And controlnet_aux uses opencv-python-headless, pin
opencv-python-headless to 4.8.0.74 too.
### Motivation and Context
### Description
For no, CoreML only support run mlmodels on CPU/ALL, However, sometimes
CPU_GPU would be faster a lot.
We support the option to select different hardware to boost performance
in this PR.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
---------
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
### Description
Change the hipify step to remove the -roc option to hipify-perl. This
will prefer hipblas over rocblas. rocblas can still be called directly
such as in TunableOp.
### Motivation and Context
hip interfaces are preferred over roc for porting from cuda to hip.
Calling roc interfaces is meant for ROCm-specific enhancements or
extensions.
- Added a microbenchmark for the `LayerNormalization` MLFloat16 support
added in https://github.com/microsoft/onnxruntime/pull/22063.
- Updated the `LayerNormalization` MLFloat16 implementation to improve
the latency.
```
----------------------------------------------------------------------------------------------
Original MLFloat16 support Time CPU Iterations
----------------------------------------------------------------------------------------------
BM_LayerNormalization<MLFloat16, float>/1/real_time 15599 us 15625 us 47
BM_LayerNormalization<MLFloat16, float>/1/real_time 14714 us 14824 us 39
BM_LayerNormalization<MLFloat16, float>/1/real_time 14634 us 14688 us 50
----------------------------------------------------------------------------------------------
Updated MLFloat16 support Time CPU Iterations
----------------------------------------------------------------------------------------------
BM_LayerNormalization<MLFloat16, float>/1/real_time 7276 us 7254 us 84
BM_LayerNormalization<MLFloat16, float>/1/real_time 6820 us 6720 us 93
BM_LayerNormalization<MLFloat16, float>/1/real_time 6840 us 6882 us 84
```
1. Add python 3.13 to our python packaging pipelines
2. Because numpy 2.0.0 doesn't support thread free python, this PR also
upgrades numpy to the latest
3. Delete some unused files.
### Description
<!-- Describe your changes. -->
This PR further optimizes matmulnbits specially for iGPUs. The phi3 demo
becomes ~12 tokens/second from ~8 tokens on iGPUs.
Some todos:
1. Make the optimization more general, Remove the blockSize = 32
limitation.
2. Tune the parameter, such as workgroupSize, components size (currently
only support components = 1), to see the performance change.
Bumps [cookie](https://github.com/jshttp/cookie) and
[socket.io](https://github.com/socketio/socket.io). These dependencies
needed to be updated together.
Updates `cookie` from 0.4.2 to 0.7.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/jshttp/cookie/releases">cookie's
releases</a>.</em></p>
<blockquote>
<h2>v0.7.2</h2>
<p><strong>Fixed</strong></p>
<ul>
<li>Fix object assignment of <code>hasOwnProperty</code> (<a
href="https://redirect.github.com/jshttp/cookie/issues/177">#177</a>)
bc38ffd</li>
</ul>
<p><a
href="https://github.com/jshttp/cookie/compare/v0.7.1...v0.7.2">https://github.com/jshttp/cookie/compare/v0.7.1...v0.7.2</a></p>
<h2>0.7.1</h2>
<p><strong>Fixed</strong></p>
<ul>
<li>Allow leading dot for domain (<a
href="https://redirect.github.com/jshttp/cookie/issues/174">#174</a>)
<ul>
<li>Although not permitted in the spec, some users expect this to work
and user agents ignore the leading dot according to spec</li>
</ul>
</li>
<li>Add fast path for <code>serialize</code> without options, use
<code>obj.hasOwnProperty</code> when parsing (<a
href="https://redirect.github.com/jshttp/cookie/issues/172">#172</a>)</li>
</ul>
<p><a
href="https://github.com/jshttp/cookie/compare/v0.7.0...v0.7.1">https://github.com/jshttp/cookie/compare/v0.7.0...v0.7.1</a></p>
<h2>0.7.0</h2>
<ul>
<li>perf: parse cookies ~10% faster (<a
href="https://redirect.github.com/jshttp/cookie/issues/144">#144</a> by
<a href="https://github.com/kurtextrem"><code>@kurtextrem</code></a>
and <a
href="https://redirect.github.com/jshttp/cookie/issues/170">#170</a>)</li>
<li>fix: narrow the validation of cookies to match RFC6265 (<a
href="https://redirect.github.com/jshttp/cookie/issues/167">#167</a> by
<a href="https://github.com/bewinsnw"><code>@bewinsnw</code></a>)</li>
<li>fix: add <code>main</code> to <code>package.json</code> for rspack
(<a href="https://redirect.github.com/jshttp/cookie/issues/166">#166</a>
by <a
href="https://github.com/proudparrot2"><code>@proudparrot2</code></a>)</li>
</ul>
<p><a
href="https://github.com/jshttp/cookie/compare/v0.6.0...v0.7.0">https://github.com/jshttp/cookie/compare/v0.6.0...v0.7.0</a></p>
<h2>0.6.0</h2>
<ul>
<li>Add <code>partitioned</code> option</li>
</ul>
<h2>0.5.0</h2>
<ul>
<li>Add <code>priority</code> option</li>
<li>Fix <code>expires</code> option to reject invalid dates</li>
<li>pref: improve default decode speed</li>
<li>pref: remove slow string split in parse</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="d19eaa1a2b"><code>d19eaa1</code></a>
0.7.2</li>
<li><a
href="bc38ffd0ea"><code>bc38ffd</code></a>
Fix object assignment of <code>hasOwnProperty</code> (<a
href="https://redirect.github.com/jshttp/cookie/issues/177">#177</a>)</li>
<li><a
href="cf4658f492"><code>cf4658f</code></a>
0.7.1</li>
<li><a
href="6a8b8f5a49"><code>6a8b8f5</code></a>
Allow leading dot for domain (<a
href="https://redirect.github.com/jshttp/cookie/issues/174">#174</a>)</li>
<li><a
href="58015c0b93"><code>58015c0</code></a>
Remove more code and perf wins (<a
href="https://redirect.github.com/jshttp/cookie/issues/172">#172</a>)</li>
<li><a
href="ab057d6c06"><code>ab057d6</code></a>
0.7.0</li>
<li><a
href="5f02ca8768"><code>5f02ca8</code></a>
Migrate history to GitHub releases</li>
<li><a
href="a5d591ce84"><code>a5d591c</code></a>
Migrate history to GitHub releases</li>
<li><a
href="51968f94b5"><code>51968f9</code></a>
Skip isNaN</li>
<li><a
href="9e7ca51ade"><code>9e7ca51</code></a>
perf(parse): cache length, return early (<a
href="https://redirect.github.com/jshttp/cookie/issues/144">#144</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/jshttp/cookie/compare/v0.4.2...v0.7.2">compare
view</a></li>
</ul>
</details>
<details>
<summary>Maintainer changes</summary>
<p>This version was pushed to npm by <a
href="https://www.npmjs.com/~blakeembrey">blakeembrey</a>, a new
releaser for cookie since your current version.</p>
</details>
<br />
Updates `socket.io` from 4.7.5 to 4.8.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/socketio/socket.io/releases">socket.io's
releases</a>.</em></p>
<blockquote>
<h2>socket.io-client@4.8.0</h2>
<h3>Features</h3>
<h4>Custom transport implementations</h4>
<p>The <code>transports</code> option now accepts an array of transport
implementations:</p>
<pre lang="js"><code>import { io } from "socket.io-client";
import { XHR, WebSocket } from "engine.io-client";
<p>const socket = io({
transports: [XHR, WebSocket]
});
</code></pre></p>
<p>Here is the list of provided implementations:</p>
<table>
<thead>
<tr>
<th>Transport</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>Fetch</code></td>
<td>HTTP long-polling based on the built-in <code>fetch()</code>
method.</td>
</tr>
<tr>
<td><code>NodeXHR</code></td>
<td>HTTP long-polling based on the <code>XMLHttpRequest</code> object
provided by the <code>xmlhttprequest-ssl</code> package.</td>
</tr>
<tr>
<td><code>XHR</code></td>
<td>HTTP long-polling based on the built-in <code>XMLHttpRequest</code>
object.</td>
</tr>
<tr>
<td><code>NodeWebSocket</code></td>
<td>WebSocket transport based on the <code>WebSocket</code> object
provided by the <code>ws</code> package.</td>
</tr>
<tr>
<td><code>WebSocket</code></td>
<td>WebSocket transport based on the built-in <code>WebSocket</code>
object.</td>
</tr>
<tr>
<td><code>WebTransport</code></td>
<td>WebTransport transport based on the built-in
<code>WebTransport</code> object.</td>
</tr>
</tbody>
</table>
<p>Usage:</p>
<table>
<thead>
<tr>
<th>Transport</th>
<th>browser</th>
<th>Node.js</th>
<th>Deno</th>
<th>Bun</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>Fetch</code></td>
<td>✅</td>
<td>✅ (1)</td>
<td>✅</td>
<td>✅</td>
</tr>
<tr>
<td><code>NodeXHR</code></td>
<td></td>
<td>✅</td>
<td>✅</td>
<td>✅</td>
</tr>
<tr>
<td><code>XHR</code></td>
<td>✅</td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td><code>NodeWebSocket</code></td>
<td></td>
<td>✅</td>
<td>✅</td>
<td>✅</td>
</tr>
<tr>
<td><code>WebSocket</code></td>
<td>✅</td>
<td>✅ (2)</td>
<td>✅</td>
<td>✅</td>
</tr>
<tr>
<td><code>WebTransport</code></td>
<td>✅</td>
<td>✅</td>
<td></td>
<td></td>
</tr>
</tbody>
</table>
<p>(1) since <a
href="https://nodejs.org/api/globals.html#fetch">v18.0.0</a>
(2) since <a
href="https://nodejs.org/api/globals.html#websocket">v21.0.0</a></p>
<p>Added in <a
href="f4d898ee96">f4d898e</a>
and <a
href="b11763beec">b11763b</a>.</p>
<h4>Test each low-level transports</h4>
<p>When setting the <code>tryAllTransports</code> option to
<code>true</code>, if the first transport (usually, HTTP long-polling)
fails, then the other transports will be tested too:</p>
<pre lang="js"><code>import { io } from "socket.io-client";
</tr></table>
</code></pre>
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="d0fc720420"><code>d0fc720</code></a>
chore(release): socket.io@4.8.0</li>
<li><a
href="4a0555c671"><code>4a0555c</code></a>
chore(release): socket.io-client@4.8.0</li>
<li><a
href="2b60df18a8"><code>2b60df1</code></a>
chore(release): engine.io@6.6.1</li>
<li><a
href="d4cb375856"><code>d4cb375</code></a>
ci: ignore tests when publishing to npm</li>
<li><a
href="c251ae7ba7"><code>c251ae7</code></a>
chore(release): engine.io-client@6.6.1</li>
<li><a
href="8a2f5a3da0"><code>8a2f5a3</code></a>
fix(eio-client): move 'offline' event listener at the top</li>
<li><a
href="b04fa64365"><code>b04fa64</code></a>
fix(sio): allow to join a room in a middleware (uws)</li>
<li><a
href="7085f0e3e4"><code>7085f0e</code></a>
refactor(sio-client): mangle private attributes</li>
<li><a
href="4f66708210"><code>4f66708</code></a>
chore(sio-client): use babel loose mode when transpiling classes</li>
<li><a
href="1a95db2145"><code>1a95db2</code></a>
chore(sio-client): add a script to compute the bundle size</li>
<li>Additional commits viewable in <a
href="https://github.com/socketio/socket.io/compare/socket.io@4.7.5...socket.io@4.8.0">compare
view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
Dependabot will merge this PR once CI passes on it, as requested by
@fs-eire.
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/microsoft/onnxruntime/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
### Description
<!-- Describe your changes. -->
With this optimization, 96 MultiHeadAttention|Transpose ops in phi3
disappear. Phi3 becomes 113 tokens from 107 tokens on my dGPUs.
The optimization mainly skips the transpose op if one of the transposed
dims is 1. Reshape is enough.
### Description
Request and create DML EP and its data transfer.
Use to copy on device.
The PR includes changes to fix issues in DML provider.
### Motivation and Context
This enables Lora users to run it with DML which is important for GenAI.
Co-authored-by: @PatriceVignola
---------
Co-authored-by: Patrice Vignola <vignola.patrice@gmail.com>
### Description
Support OV2024.4
Refactor tensor initialization check for external weights
Support loading OV Config
OVEP: Tensor Caching fix, Fix accuracy issues
Refactor device memory implementation to make it more generic
### Motivation and Context
The changes are required to fix accuracy issues, support loading of OV
config, support OV2024.4
---------
Co-authored-by: Eric Crawford <eric.r.crawford@intel.com>
Co-authored-by: saurabhkale17 <saurabh1.kale@intel.com>
Co-authored-by: Javier E. Martinez <javier.e.martinez@intel.com>
Co-authored-by: sfatimar <sahar.fatima@intel.com>
Co-authored-by: ankitm3k <ankit.maheshkar@intel.com>
Co-authored-by: Preetha Veeramalai <preetha.veeramalai@intel.com>
Co-authored-by: n1harika <niharika.sathish@intel.com>
Co-authored-by: jatinwadhwa921 <110383850+jatinwadhwa921@users.noreply.github.com>
- Work around Xcode 16 iOS test build issue: `error: Multiple commands produce '.../PlugIns'`.
- Fix link error in iOS static framework test.
- Update build.py to check for the right kind of build before running iOS tests on the simulator.
- Update Xcode 16 build images to 'macos-15' because that's the only image that will have Xcode 16 soon. See https://github.com/actions/runner-images/issues/10703.
### Description
Add a new pipeline to publish ROCM package to ADO
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Test Link
https://dev.azure.com/aiinfra/Lotus/_build?definitionId=1615
### Description
This PR fixes a bug when the ONNX checker is called while exporting
Whisper for FP16 CUDA with optional flags.
### Motivation and Context
Sometimes, the ONNX checker raises an error depending on the optional
flags passed. By wrapping the ONNX checker in a try-except, the
conversion can continue even if the checker fails.
### Description
Add additonal gfx targets for AMD GPU support
### Motivation and Context
Required to integrate mainline onnxruntime support for AMD GPUs
---------
Co-authored-by: Stefan Sokolovic <stsokolo@amd.com>
Co-authored-by: Jeff Daily <jeff.daily@amd.com>
[TensorRT-Model-Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer)
have a implementation for INT4 AWQ. Adding the support in onnxruntime
tools to quantized the models with TensorRT-Model-Optimizer
### Description
* Add digital signature to dll files in jar files.
* Jar file names: onnxruntime-{version}.jar,
onnxruntime_gpu-{version}.jar
### Motivation and Context
#19204
### Description
<!-- Describe your changes. -->
flatbuffers always write data in LE and it is automatically traslated
to/from BE as needed,
but only if we use proper accessors. This would work for shape.
However, we store parameters as bytes, so we need to swap bytes as
needed for BE.
### Motivation and Context
Address https://github.com/microsoft/onnxruntime/issues/22364
### Description
Aallows alpha, beta and rc version releases to Maven for Android
artifacts.
### Motivation and Context
Helpful to release rc versions or test artifacts to Maven for testing.
For example, a new QNN android package is being released and it will be
nice to test the RC version for dependencies before release
## Future Work
Allow RC version for all Maven artifacts.
### Description
<!-- Describe your changes. -->
Pick up onnx/onnx#6010 to support EinSum shape inference
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
This change allows EinSum operator's output shape to be inferenced so
that it can run on accelerators.
### Description
Pre built QNN Android package
### Future Work
1. Setting up CI with Browserstack- onnxruntime_tests and Android test
2. ESRP Release to Maven
### Description
In TensorRT 10.5, the APIs `platformHasFastFp16` and
`platformHasFastInt8` have been deprecated.
Ignore these deprecation warnings.
Signed-off-by: Kevin Chen <kevinch@nvidia.com>
### Description
Update segment anything 2 benchmark script:
(1) Fix cuda graph in benchmark. Make sure --use_cuda_graph takes effect
and random_inputs() generates according to the dtype of the model.
(2) Add a parameter to enable profiling.
(3) Use latest cuda 12.6.2 and cudnn 9.5.
(4) Update README.md.
### Motivation and Context
Previous, --use_cuda_graph does not take effect. This fixes the
benchmark.
### Description
Add SetEpDynamicOptions and Remove workload_type from run/session
options.
### Motivation and Context
Added SetEpDynamicOptions as a dynamic way of changing EP settings even
in the middle of a Run
Using workload_type run/session options to set Efficient/Default mode
for workloads does not cover all the scenarios and can lead to priority
inversions. Working on a new API to support setting Efficient/Default
mode for workloads.
---------
Co-authored-by: Luis E. Pena <luispena@microsoft.com>
### Description
Resolve#21976 .
ABSL generally does not have forward/backward compatibility. Our code is
only compatible with one fixed LTS version. So it's important to fix the
version number there when using find_package to detect an installed
version.
### Description
It runs after "Python-CUDA-Packaging-Pipeline" that runs on a CPU
machine that skipped all tests.
This testing pipeline is for doing the tests.
This fixes a bug found by libfuzzer:
LayerNormalization third input (beta) is optional. The following code
has potential out of bound access if the input is not available:
```
NodeArg* beta = layer_norm_node.MutableInputDefs()[2];
```
This adds a check to ensure the third input exists before fusion.
[AB#49036](https://aiinfra.visualstudio.com/6a833879-cd9b-44a4-a9de-adc2d818f13c/_workitems/edit/49036)
### Description
* Add a few arguments CUDA_VERSION, CUDNN_VERSION, OS, GIT_COMMIT,
GIT_BRANCH and ONNXRUNTIME_VERSION to the Dockerfile.cuda to allow for
more flexibility in the build process.
* Update README.md to include the new arguments and their usage.
* Output labels to image so that it is easy to inspect the image.
Available CUDA versions for ubuntu 24.04 can be found
[here](https://hub.docker.com/r/nvidia/cuda/tags), and available CUDNN
versions can be found
[here](https://pypi.org/project/nvidia-cudnn-cu12/#history). Example
command line to build docker image:
```
docker build -t onnxruntime-cuda --build-arg CUDA_VERSION=12.6.1 \
--build-arg CUDNN_VERSION=9.5.0.50 \
--build-arg GIT_BRANCH=$(git rev-parse --abbrev-ref HEAD) \
--build-arg GIT_COMMIT=$(git rev-parse HEAD) \
--build-arg ONNXRUNTIME_VERSION=$(cat ../VERSION_NUMBER) \
-f Dockerfile.cuda ..
```
Example labels from `docker inspect onnxruntime-cuda`:
```
"Labels": {
"CUDA_VERSION": "12.6.1",
"CUDNN_VERSION": "9.5.0.50",
"maintainer": "Changming Sun <chasun@microsoft.com>",
"onnxruntime_git_branch": "main",
"onnxruntime_git_commit": "bc84958dcef5c6017ae58085f55b669efd74f4a5",
"onnxruntime_version": "1.20.0",
"org.opencontainers.image.ref.name": "ubuntu",
"org.opencontainers.image.version": "24.04"
}
```
### Motivation and Context
https://github.com/microsoft/onnxruntime/pull/22339 has hard-coded the
cuda and cudnn versions. User might want to choose specified cuda and
cudnn version during building docker image.
Fix the QNN nuget package issue
### Description
Inside the package, folder name \runtimes\win-arm64\ was changed to \runtimes\win-ARM64\, which breaks lib copy settings in Microsoft.ML.OnnxRuntime.QNN.props.
### Motivation and Context
Fix issue: https://github.com/microsoft/onnxruntime/issues/21692
### Description
Update the commit from 59600894a2c1c18290944b83e989bfe618975230 to
1887322ed36d522409a6b805d4e7942cf76a8e40
### Motivation and Context
The new one has python 3.13.
AB#50959
This reverts commit 4e15b229a0.
Reason: We are seeing an increase in the number of deadlocks after this
PR. We have a release coming up next week and do not have enough time to
investigate the root cause, hence reverting this PR temporarily.
Moreover, this is causing an increase int he binary size.
### Description
We are seeing an [increase in the number of
deadlocks](https://github.com/microsoft/onnxruntime/pull/22315#issuecomment-2394821893)
after this PR. We have a release coming up next week and do not have
enough time to investigate the root cause, hence reverting this PR
temporarily.
### Motivation and Context
See above.
### Description
This change introduces the WebGPU EP into ONNX Runtime.
To make the PR as simple as possible, this PR excluded the following:
- C API changes for WebGPU EP
- actual implementation of WebGPU EP. Currently in this PR, WebGPU is a
stub implementation that does not register any kernel.
- Python IO Binding update
- Node.js IO Binding update
This PR now contains only 43 file changes (while the working branch
contains 130+) and hopefully this makes it easier to review.
There is going to be separated PRs for each mentioned above.
Current working branch: #21904
