### Description
The machine has multiple python installations and none of them is in
PATH. Therefore we should explicitly set python version via this task to
avoid having surprises.
### Motivation and Context
Similar to #21095
### Description
Delete RoslynAnalyzers. Use CodeQL instead.
### Motivation and Context
Now we already have CodeQL which is modern and also covers C# code. The
RoslynAnalyzers one is not in our pull request pipelines. The
"RoslynAnalyzers@2" task is outdated and needs be upgraded. I will
delete it for now since we already have CodeQL.
### Description
1. added kernel to quantize matmul B tensor to q4, and store in the same
shape as original tensor. scales and zero points are calculated as well.
scales and zero points have the same shape.
2. added kernel to transpose q4 B tensor to B tensor in MatMulNBits.
Scales and zero points are transposed as well.
####
Benchmark
<1024 x 4096 input, 64 quant block, 8 threads>:
- quantize: 23035923 ns
- transpose: 718635 ns
<1024 x 4095 input, 64 quant block, 8 threads>:
- quantize: 26759319 ns
- transpose: 1279064 ns
### Motivation and Context
The MatMulNbits tool chain current only supports converting a MatMul op
direct to MatMulNBits op. MatMulNbits op is not an ONNX standard op.
Therefore, we need the tool chain to support converting MatMul to Q/DQ
format, and later in the transform step converts DQ + MatMul to
MatMulNBits. The tensors stored in DQ are the quantized constants and
will be stored in the MatMulNBits.
### Description
change is_pod tp is_trivial
### Motivation and Context
This is commonnly needed for both linux and win c++20 upgrade.
is_trivial was introduced backed in C++11
### Description
Remove the "--enable_language_interop_ops" build flag, because the code
is incompatible with the latest numpy, and the build flag is not used
anywhere except a macOS CI pipeline. It does not seem to have a ship
plan.
### Motivation and Context
The build error was:
```
onnxruntime/core/language_interop_ops/pyop/pyop.cc:122:85: error: no member named 'elsize' in '_PyArray_Descr'
static_cast<int64_t>(PyArray_DescrFromType(type)->elsize),
~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
```
### Description
Due to a security setting change, now we need to explicitly set the
permissions. I forgot doing that when bringing the old change back.
### Motivation and Context
Now the pipeline cannot publish scanning result to Github
### Description
- Updates CI pipelines to use QNN SDK 2.23.0 by default.
- QNN SDK adds support for int64 Cast. This allows QNN EP to support
ONNX ArgMax/ArgMin/TopK operators that generate an int64 graph output.
Example translation of ArgMax:
- **ONNX**: input --> ArgMax --> output (int64)
- **QNN**: input --> ArgMax --> Cast (int32 to int64) --> output (int64)
### Motivation and Context
Update onnxruntime to use the latest QNN SDK.
Refactoring of MultiHeadAttention op
- [x] Add some checking for cross attention of pass_past_in_kv to make
sure there is no kv cache and bias.
- [x] Update interface of PackVIntoRotaryQKV so that it can be used by
SparseAttention later.
- [x] Add test cases
### Motivation and Context
To prepare the pull request for SparseAttention cpu op.
### Description
<!-- Describe your changes. -->
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
<!-- Describe your changes. -->
Remove xamarin related entries.
Update MAUI entries to net8
Remove macos entries (not required by MAUI)
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Updates missed from #21062
WebNN CPU implementation has been migrated from XNNPack to TFLite which
supports more ops. Turn on partial `cpu` supported ops which just need
the change from `false` to `true` firstly.
### Description
<!-- Describe your changes. -->
Xamarin is EOL so remove support.
The MAUI targets are EOL and need updating.
https://dotnet.microsoft.com/en-us/platform/support/policy/maui
Other cleanups:
- netcoreapp3.1 is EOL
- the net6 macos target was added in the mistaken belief that was for
MAUI mac support, but that is actually via the mac-catalyst target which
we recently added support for.
- some CIs that were using the old build setup of splitting pre-net6
targets. The ORT C# bindings csproj was updated last year and the
`PreNet6` and `SelectedTargets` properties no longer exist as they were
replaced by the simpler `IncludeMobileTargets` property.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Remove EOL components.
#21058
### Description
Fix regression caused by
https://github.com/microsoft/onnxruntime/pull/20995
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
skip default `locateFile()` when dynamic import is disabled. This allows
the file to work with bundlers to load WebAssembly file correctly if
`env.wasm.wasmPaths` is not set.
See #20643
### Description
Changes order of how we perform quantization to better support mixed
precision and fixes a bug found with parameters of inputs for int8
quantization not being correctly handled.
We now perform int8 quantization first on a full precision input model,
before then quantizing the model to fp16 for remain ops that aren't
quantized. The former case was causing us to use a low precision input
which could cause larger values to be inserted than intended to the
model when int8 quantization is perform. The symptom of this was a
failure during quantization steps.
Similar to the above input parameters were being uninitialized and
resulting in similar failure during int8 quantization.
GPU faults were intermittent but present as using uninitialized memory
created undefined behavior when we started testing more complex models
during mixed precision.
### Motivation and Context
In some cases we've seen random data and/or invalid values entering into
compiled onnx graphs. This is due to input parameters to the MIGraphX
Graph not being set correctly when mixed precision (int8 + fp16) is used
and ordering of quantization steps is causes a lower precision model to
be used to perform int8 quantization. In most cases the failure is
silent/intermittent. In some cases we've observed gpu faults due to out
of bounds values being set.
This change is required as a large input parameter to the MIGraphX graph
is initialized to a large random value, and the next operator is using
that for indexing, we get undefined behavior and a GPU fault.
### Description
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
Upgrade pybind11 to the latest as suggested by @gnought in #21063
### Motivation and Context
Recently numpy released a new version, which caused compatibility issue
between the latest numpy version and the latest ONNX Runtime version.
### Description
<!-- Describe your changes. -->
- Add check for CoreML MLProgram supported ops
- Only check usability with ORT Mobile package if requested
- this package will be deprecated so info is a) of minimal value and b)
can be confusing.
- Output more things at INFO level
- a lot of meaningful info was only output at DEBUG level. The default
INFO level is more useful
- dump full partition info at DEBUG level
- Check subgraphs fully
- CoreML can handle a subgraph
- TBD if we want to add support for adding a subgraph to the parent
graph for Loop and If nodes
- most likely will be required for simple If nodes to be performant
- Check 5D CoreML limitation
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Improve helper tools
---------
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
### Description
In CUDA case, use the cuda_home variable to set CMAKE's CUDA compiler to
a correct version of NVCC
Otherwise, an NVCC from a current PATH would be picked up, which could
be from a different version of CUDA.
### Motivation and Context
I had a case when I had main CUDA installed, and it was a version 11.8.
I wanted to build against 12.5, so I downloaded and unpacked it into a
separate directory and passed it as a `--cuda-home` parameter, however
the ONNX builder was still picking the NVCC compiler from 11.8.
This would fix the issue
https://github.com/microsoft/onnxruntime/issues/20928
cc @gedoensmax
### Description
Updates QNN EP to reject Conv/ConvTranspose/Q/DQ ops with unsupported
quantization axis values.
### Motivation and Context
Allows these unsupported operators to be handled by the CPU EP.
Fixes errors like the following:
> Node 'ConvTranspose' OpType:ConvTranspose with
domain:com.ms.internal.nhwc was inserted using the NHWC format as
requested by QNNExecutionProvider, but was not selected by that EP. This
means the graph is now invalid as there will not be an EP able to run
the node. This could be a bug in layout transformer, or in the
GetCapability implementation of the EP.
---------
Signed-off-by: adrianlizarraga <adlizarraga@microsoft.com>
WebNN Spec adds missing 64-bit integers support for `reduceL1`,
`reduceSum`, `reduceSumSquare` and `reduceProduct` ops at this
[PR](https://github.com/webmachinelearning/webnn/pull/695), which has
already been implemented in Chromium. Update corresponding data type
constraints in WebNN EP.
Besides, WebNN CPU backend currently doesn't support `uint64` and
`uint32` for these ops.
### Description
<!-- Describe your changes. -->
Fix the threshold limit
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
We use this to debug the graph after each graph transformation. But, the
const data inside the model is useless.
So, we decided to remove that information to save disk space.
Co-authored-by: Chunye Wang <chunywan@xlnx.xilinx.com>
### Description
Adds the ability for MIGraphX EP to save off or load compiled models to
save time between inferences.
Via Command line
User should be able to set the save ability with
ORT_MIGRAPHX_SAVE_COMPILED_MODEL
ORT_MIGRAPHX_SAVE_COMPILE_PATH
User should be able to set the load ability with
ORT_MIGRAPHX_LOAD_COMPILED_MODEL
ORT_MIGRAPHX_LOAD_COMPILE_PATH
via Onnxruntime API
migx_save_compiled_model
migx_save_model_name
migx_load_compiled_model
migx_load_model_name
### Motivation and Context
The motivation for this is to leverage MIGraphX's existing API to
save/load models after our compile step of graph optimization. For
larger models or models which were compiled with additional tuning
steps, this saves time after first compile and inference run, and thus
speeds up the user experience in order to encourage development.
---------
Co-authored-by: Ted Themistokleous <tedthemistokleous@amd.com>
### Description
<!-- Describe your changes. -->
The tools should really all come from the same Android NDK, so using
`shutil.which` adds potential confusion when we do a lookup for the
target program by name first due to adding `dirnames.insert(0, "")` as
the first directory entry to lookup as it will match the filename
anywhere in the current path.
That's problematic as the emulator should come from
<sdk_tools>/emulator/emulator (see
[here](https://www.stkent.com/2017/08/10/update-your-path-for-the-new-android-emulator-location.html)),
but the paths on the CI machines result in the old location of
<sdk_tools>/tools/emulator being selected. This leads to the emulator
failing to run on arm64 macOS CIs as the old emulator does not look for
the arm64 binary.
At the most you may have multiple cmdline-tools versions installed, but
if we need to support explicitly specifying a version for that path that
can be added.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Make emulator run on arm64 macOS machines.
Add some helper functions to dump 4D tensors to help debugging.
Example to use it:
(1) Change DUMP_TENSOR_LEVEL from 0 to 2 in
contrib_ops/cpu/utils/debug_macros.h to enable dumping. Without
enabling, the dumping code will not be built into ORT binary.
(2) Add a few lines to dump tensors like
```
DUMP_CPU_TENSOR_INIT();
DUMP_CPU_TENSOR("tensor name", tensor_data, dim0, dim1, dim2, dim3);
```
Changes:
- [x] Add functions to dump 4D int32/int64/float/half tensors in CPU
- [x] Add functions to dump 4D int32/int64 tensors in CUDA
- [x] Change namespace (remove .transformers from namespace, and move
files to utils directory)
### Description
- Fixes compilation error for "reduced operator" builds with no FP16
kernels and `MLAS_F16VEC_INTRINSICS_SUPPORTED` enabled.
- Fixes linker error for "reduced operator" builds with QNN EP by
excluding QNN EP unit tests. QNN EP unit tests require CPU EP operator
implementations to evaluate accuracy.
### Motivation and Context
Need to be able to build a reduced operator build with QNN EP. See
https://github.com/microsoft/onnxruntime/blob/main/docs/Reduced_Operator_Kernel_build.md
The following example operator config file causes a compilation error
when either `MLAS_F16VEC_INTRINSICS_SUPPORTED` is defined or QNN EP is
enabled.
```
# reduced_op_config.txt
ai.onnx;12;Add
```
```shell
python tools\ci_build\build.py --include_ops_by_config reduced_op_config.txt --config Debug --build_wheel --build_shared_lib --skip_tests --build_dir build --parallel --use_qnn --qnn_home '<QNN_ROOT_DIR>'
```
Avoid using command line flags to pass in CMAKE_PREFIX_PATH. Use
environment variables instead.
Because, otherwise the value of CMAKE_PREFIX_PATH could get encoded
twice. For example, if the prefix is `C:\a\root`, then in
tools/ci_build/github/windows/helpers.ps1 we set it in Env:CMAKE_ARGS
which will be consumed by ONNX. Then when ONNX get it and decoded it,
ONNX will get `C:aroot` instead. Then because the path doesn't exist,
the CMAKE_PREFIX_PATH couldn't take effect when the script installs
ONNX. This PR fixes the issue.
The issue got discovered when I tried to upgrade cmake to a newer
version. Now our Windows CPU CI build pipeline uses cmake 3.27. In the
main branch even the CMAKE_PREFIX_PATH setting does not work, cmake
still can find protoc.exe from the directories. However, starting from
3.28 cmake changed it. With the newer cmake versions the find_library(),
find_path(), and find_file() cmake commands no longer search in
installation prefixes derived from the PATH environment variable.
### Release backward inputs per static graph ref count
For the output buffer marked as external output:
1. Remove the additional ref count we used for avoiding reusing buffer.
Instead, when we find reuse input/output buffer, we will make sure the
reused buffer not not generated by nodes that has external outputs.
2. Remove the ref count of pybind feed inputs, which exists all the time
until the run_backward completed. Instead, passing a mutuble feeds, and
we clean the feeds vector once that is copied into session states and
not needed any more before run the graph sequencentially.
#### Before the change:
One of the backward inputs is 3.9GB, it lives until the backward ends.
#### With the change:
The 3.9GB is released when the last node depending on that tensor
completed.
Be noted: the peak did not change though, we have more work to do to
reduce on the peak.
#### Others
It is found there are few tests that were updated to use incorrect
expected values in previous code refactoring
a81faee41e (diff-9e8fbae7d3dff24106cd17564949f320e943cb3048eae07813c7de144f140419L382).
This PR tries to fix them back, and I think now all test cases are back
to normal.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Several changes to remove warnings or errors on CPU benchmark and other
benchmarks.
- Phi-3 codes doesn't require auth token, but trust_remote_code flag.
Add "--trust" to enable only trust_remote_code.
- Phi3 models are not working with sdpa and needs to be run with eager
mode.
- Fix CPU io binding error with null device id and element_type
mismatch.
- use_buffer_share only when engine is ort.
### Description
<!-- Describe your changes. -->
Check if the onnxruntime_vitisai_ep.dll already linked. If yes, don't
use dlopen.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
If a executable is linked directly with onnxruntime_vitisai_ep.dll.
dlopen would cause two DLL with same content loaded. So we have to check
that case.
Linux can detect such situation automatically.
### Description
Fix some typos in Vitis AI EP.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Co-authored-by: liumingyue <mingyue@xilinx.com>
