### Description
- Add new Ops: Ceil, Exp, Identity, Reciprocal, Tan.
- Set MinSupportedOpSet for unary Ops.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Support more Ops for other models.
The legacy optimization attribute "consumed_inputs" is not supported in
WebNN EP.
### Description
<!-- Describe your changes. -->
Publish E2E test logs on build failure too.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Get more information about intermittent test failures.
### Description
Added support for ReduceL1, ReduceL2, ReduceMean, ReduceMin, ReduceMax,
ReduceSum, ReduceLogSum, ReduceLogSumExp, ReduceProd and
ReduceSquareSum.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
---------
Co-authored-by: Satya Jandhyala <sajandhy@microsoft.com>
Co-authored-by: guschmue <guschmue@microsoft.com>
### Enhance StatisticsSubscriber
There are few improvements for `StatisticsSubscriber`:
- Reduce peak memory impact for tensors (having many many many elements,
consuming too much GPU memory, causing original recipe run failed with
OOM), by split the statistics into two phases (split into buckets, and
merge result across buckets).
- Allow dump intermediate tensors. Originally only nn.Module forward()'s
return value are dumped, there are requirements we want to inspect some
specific intermediate tensor in the forward() function, now we support
it.
- Add documents for collecting dumps on multiple ranks
Docs link on this branch for better view:
https://github.com/microsoft/onnxruntime/blob/pengwa/conv_tool_v2/docs/ORTModule_Convergence_Notes.md
---------
Co-authored-by: mindest <30493312+mindest@users.noreply.github.com>
### Description
A few QDQ tests failed on XNNPACK EP.
The reason should be the range of input_data doesn't fit for scale and
zero_point.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
<!-- Describe your changes. -->
This
[PR](e726151b5c (diff-6957596681c25d78e7f3f56485f307fb7e66369309523240209a62c8fa21646b))
introduces a missing registration of Identity operator for version
greater than 14.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
It broke the CANN CI. I added the registration of identity operator.
### Description
NNAPI Doesn't support the indices input of Gather to be a scalar.
To workaround it.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
<!-- Describe your changes. -->
refactor tensor type in onnxruntime-common.
### Motivation and Context
There major motivation is that I am doing a local change to address the
API part of #15312. And I am doing a refactoring of onnxruntime-common
anyway (#15772).
The `tensor.ts` and `tensor-impl.ts` are too large, so I split contents
into multiple files to make the type declarations clearer.
The original target of this change is for API only ( ie. do not refactor
any implementation.). However, there are a few type/implementation
inconsistencies so I also made minimal changes to fix them.
### Changes
- extract `TensorUtils` for non-template interfaces
- extract `TensorFactory` for all overloads of `Tensor.fromImage()`
- refactor options type that used for `Tensor.fromImage()`
- fix JSDoc comments to make option descriptions consistent with actual
type declarations
- fix an inconsistency for `options.format` and `options.bitmapFormat`;
change all `bitmapFormat` to `format`
- extract `ConversionUtils` for `tensor.toDataURL()` and
`tensor.toImageData()`
- put implementations into multiple files from `tensor-impl.ts`
- fix a bug that cause unittest fail. put comments for future fix.
### Description
Add an API for users to get version of current package. example usage:
```js
import { env } from 'onnxruntime-node';
console.log(env.versions.node); // output "1.16.0"
```
```js
import { env } from 'onnxruntime-web';
console.log(env.versions.web); // output "1.16.0"
console.log(env.versions.common); // output "1.16.0"
console.log(env.versions.node); // output "undefined"
```
#16156
### Description
1. Updated Mac package workflow for easily debugging.
2. Changed Archive type from tgz to zip since zip is supported by ESRP.
3. .../dylib.dSYM/Contents/Resources/DWARF/libonnxruntime.1.16.0.dylib
is a debug symbol file, so it couldn't be signed.
### Motivation and Context
It‘s required from VS code.
Mac binaries in nuget should be signed
### Description
Correctly sets padding when the `auto_pad` attribute is specified for
Conv operator.
### Motivation and Context
Needed to correctly translate ONNX Conv to QNN Conv2d.
### Description
<!-- Describe your changes. -->
SetThreadDescription isn't available in an Azure App Service sandbox.
#15219 removed a check that it was available, making it a hard
dependency. When it's not available the dll load fails with a 'procedure
not found' error.
Add back the check.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
#15375 - although note this has nothing to do with the original issue.
This is just for
https://github.com/microsoft/onnxruntime/issues/15375#issuecomment-1579464889
### Description
Add model description into context binary file metadata for validation
### Motivation and Context
Dump more information for validation
---------
Co-authored-by: Adrian Lizarraga <adlizarraga@microsoft.com>
### Description
Fix an issue for Conv with dynamic weights
Root cause:
Conv op builder create the weight input tensor with wrong name. With dynamic weight, Transpose node is inserted. Conv op builder should use the new name which is Transpose output. It cause the weight producer has wrong output shape.
### Description
<!-- Describe your changes. -->
Implement `dispose` react native method.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Currently we are not able to release the memory used by model in JS
runtime if we don't want to use it anymore, we can do that only by
reload app on debug or restart app on release.
### Description
Adds tests for operators that return error 1002
(QNN_COMMON_ERROR_MEM_ALLOC) when the call to graphFinalize() fails.
This seems to happen for large input sizes.
Operators:
- Sub
- Div
- Conv
- MaxPool
### Motivation and Context
This documents bugs that need to be addressed with unit tests.
### Description
1. Use IAllocatorUniquePtr to replace BufferUniquePtr. It will ensure
the deleter is always right.
2. Change some std::unique_ptr to std::optional
3. Bypass Arena allocator when allocating the prepack buffers for mlas.
In this special case, Arena doesn't help any. And this change is just an
internal implementation change, it doesn't affect our public interface.
### Description
This PR adds flags for exporting Whisper with vocab masks for logits
processing. This PR also sets `input_features` back to FP32 precision
for the user and casts `input_features` to FP16 precision when needed.
### Motivation and Context
This helps enable specific logits processing for the exported Whisper
model.
### Description
google/re2 [was
switched](49d776b9d2)
to absl::string_view in version 2023-06-02.
As `absl::string_view` is a drop-in replacement for `std::string_view`
it does not have `as_string()` method.
This PR ensures the forward compatibility with the newest versions of
re2 library.
### Description
<!-- Describe your changes. -->
Update NNAPI Softmax to coerce to 2D when opset is < 13. This prevents
the layout change to NHWC from breaking the implementation, as well as
making it work correctly when the ONNX node's axis != 1.
Add check for opset 13+ that axis is inner-most dimension as we don't
currently handle any other value correctly.
Update tests to add model to check NHWC layout, as well as 4D tests. We
didn't notice the issues with the NNAPI EP as it was only processing
input shapes that were 2D or 4D (which was overly restrictive as well).
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
#15949
### Description
We should avoid using the macro since the value of the macro is
inaccurate. For example, our prebuilt packages are built with CUDA 11.8
but people may run the binaries with CUDA 11.4. (The minimal CUDA version we support is CUDA 11.4)
A runtime function should be used to determine CUDA version. Like:
```C++
int cuda_runtime_version = 0;
CUDA_CALL_THROW(cudaRuntimeGetVersion(&cuda_runtime_version));
ORT_ENFORCE(cuda_runtime_version >= 11040, "ONNX Runtime needs cuda runtime higher than 11.4");
```
### Description
Implement Objective-C binding for `ORTCheckPoint`. Additionally,
- Modify `onnxruntime_objectivec.cmake` to only include training header
and sources when training flag is enabled
- Enable objective-c binding for `orttraining-mac-ci-pipeline`
### Motivation and Context
This PR is part of implementing Objective-C bindings for training API.
It implements objective-c binding for ORTCheckPoint class. The
objective-C API closely resembles the C++ API.
**Note**: The test for saving checkpoint is skipped as it requires use
of training session. It will be added when the objective-c binding for
`ORTTrainingSession` is added.
### Description
The name of the flag we set when compiling the JNI binding to enable the CoreML EP changed at some point in the past. This PR fixes it by updating the flag in the JNI. I also added a quick smoke test for the CoreML provider to make sure it doesn't crash and can be enabled.
### Motivation and Context
All the EPs should work as expected in Java. Fixes#16230.
- Fix flatbuffers flatc warning, unused-but-set-variable.
- Address `-Wshorten-64-to-32` warnings (fix in our code, allow in dependencies' code).
- Update CI builds to use Xcode 14.3.
- Update minimum iOS version to 12.0.
- Update Mac hosted agents to MacOS 13 where possible.
### Description
Increases allowable accuracy tolerance for specific Conv op test on QNN
CPU backed (Windows x64).
### Motivation and Context
Allow QNN NuGet pipeline to run. PR
https://github.com/microsoft/onnxruntime/pull/15975 introduced a failing
test on Windows x64.
We implemented a number of new ops and data types to support running
segment anything model on Chromium WebNN DML backend (POC) in a forked
branch https://github.com/honry/onnxruntime/tree/stable-diffusion
In this PR, we migrate the changes in the forked branch to main branch,
includes:
- 22 new ops
- New tensor data types: bool, int32, uint32, uint64, int64, float16 (As
JavaScript hasn't shipped Float16Array, we use Uint16Array as a
workaound)
- Handle empty input tensors and duplicated outputs
- Fixed some nits
1. Add new test data GetSelfAttentionData_WithPastAndPresent_HeadSize8_NoMask_NoRelPosBias, also added non-biased data
2. Add new test data GetCrossAttentionData_DiffSequenceLengths_HeadSize8, also added non-biased data
3. Disabled the new tests for CUDA EP due to qkv is not correctly transposed.
### Description
The tensor creation code now allows the creation of boolean tensors from
non-direct `ByteBuffer` instances. It previously only allowed them from
arrays and direct `ByteBuffer` instances and this fixes that
inconsistency. The boolean tensor test has been updated to cover all
three cases.
### Motivation and Context
Fixes#15509.
MIGraphX CI
- Change docker container user name to `onnxruntimedev`
ROCm CI
- Build docker image every job instead of using prebuild image.
- Every job create a container with only one GPU with command `docker
run -it --device=/dev/kfd --device=/dev/dri/renderDxxx`
- Remove tests that are unstable or use outdated interfaces.
- Enable training ortmodule test.
### Description
<!-- Describe your changes. -->
Detect fake tensor mode if it has already been created. Follows this
example in pytorch:
86c7652503/torch/_inductor/compile_fx.py (L280)
### Motivation and Context
As of torch nightly 6/2/23, when trying to run a torch dynamo graph on
the ORT backend, we observe
```
E torch._dynamo.exc.BackendCompilerFailed: backend='compiler_fn' raised:
E AssertionError: Mixing fake modes NYI
E
E
E You can suppress this exception and fall back to eager by setting:
E import torch._dynamo
E torch._dynamo.config.suppress_errors = True
```
The issue is that `ort_backend.py` creates a new fake tensor mode even
though one has already been created by torch.
### Description
The proposed fix is to store the result of AsBlockSparse() in a variable
to ensure the object isn't destroyed until the end of the current scope.
### Motivation and Context
"own_buffer_tensor" is a temporary object that is destroyed at the end
of the expression and causes a compile error.
### Description
1. Avoid taking dependency on dl.fedoraproject.org
The website is not very stable. Our build pipelines often fail to fetch
packages from there.
2. Update manylinux to the latest version
Fixes#13119 top concerns by
* using `onnxruntime::AllocatorDefaultAlloc` instead of `malloc`
* set `MLAS_DEFAULT_PREFERRED_BUFFER_ALIGNMENT=64` which cascades that
value
to several members and functions not directly related to MLAS.
### Motivation and Context
* Fixes#13119 top concerns. Otherwise, alignment is to 16 bytes circa
1990s 👴
* Does not yet enable flexible alignment. Instead fixed at 64 (64 x 8
bits=512 bits) for modern NN hardware like AVX-512
### Description
- Updates QDQ transformer to handle QDQ logical operators (Equal, Less,
LessOrEqual, Greater, GreaterOrEqual).
- Expects 2 DQ inputs and no Qs in the output, which is boolean.
### Motivation and Context
This is needed to enable QDQ models with logical comparison operators to
run on QNN EP.
### Description
1. Add UT for cached Qnn context binary
2. Minor change: set model path to "" if model_path is not available
since the model could be loaded from buffer instead of Onnx file
### Motivation and Context
support more scenario
---------
Co-authored-by: Adrian Lizarraga <adlizarraga@microsoft.com>
### Description
1. Add a Memory Profiling build job
2. Remove no absl build job since the feature will be removed
3. Simplify post-merge-jobs.yml by unifying the pool names
### Motivation and Context
To catch build errors in #16124
### Description
This PR adds an implementation of the Squeeze operator to WebGPU JSEP.
The implementation follows the [operator
schema](https://github.com/onnx/onnx/blob/main/docs/Operators.md#Unsqueeze).
To implement the `Unsqueeze` operator in the same fashion as the
`Squeeze`, I added the `ComputeOutputShape()` method to the
`UnsqueezeBase` class and made some slight modifications. Please let me
know if it is a bad idea and if I should move this method to the JS
implementation.
I also uncommented test case lines in the `suite-test-list.jsonc` file
for both Squeeze and Unsqueeze operators following @hariharans29's
[comment](https://github.com/microsoft/onnxruntime/pull/16024#issuecomment-1565113633).
### How was it tested
1. I created a model with only one operator:
```Python
import onnx.helper
node = onnx.helper.make_node(
"Unsqueeze",
inputs=["T", "axes"],
outputs=["y"],
)
graph = onnx.helper.make_graph([node], "test", [onnx.helper.make_tensor_value_info("T", 1, [3, 4, 5]), onnx.helper.make_tensor_value_info("axes", 7, [2])], [onnx.helper.make_tensor_value_info("y", 1, [3, 1, 4, 5, 1])])
onnx.save(onnx.helper.make_model(graph), "unsqueeze.onnx")
```
2. I compiled the runtime using @fs-eire's
[instructions](https://gist.github.com/fs-eire/a55b2c7e10a6864b9602c279b8b75dce).
3. I ran the test models in the browser using this minimal setup:
```HTML
<html>
<script src=".\dist\ort.webgpu.min.js"></script>
<script>
async function run() {
const session = await ort.InferenceSession.create('unsqueeze.onnx', {executionProviders: ['webgpu']});
console.log(session);
const input = new ort.Tensor('float32', new Float32Array(60), [3, 4, 5]);
const dim = new ort.Tensor('int64', [1n, 4n], [2]);
const output = await session.run({ "T": input, "axes": dim });
console.log(output);
}
run();
</script>
</html>
```
### Motivation and Context
Improve operator coverage for WebGPU JSEP.
