### Description
It's the prerequisite step of reducing complexity of current zip-nuget
pipeline.
Some packaging tasks could be cut from the most complex nuget pipline
and easily be published
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
Disable using CoreML ML Program for a matmul where one of the inputs is
1D as the CoreML implementation appears to be broken. See
https://github.com/apple/coremltools/issues/2263
Add some debugging notes.
### Motivation and Context
Fix failing test on macos-14.
### Description
<!-- Describe your changes. -->
-It is an initial PR for VSINPU execution provider
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
- For support VeriSilicon hardware
- TIM-VX(Tensor Interface Module)
(https://github.com/VeriSilicon/TIM-VX) is an integrated software
solution by Verisilicon for our hardware(A311D/i.MX 8M Plus etc.)
design, it is easy to use Verisilicon’s hardware by simply connecting
onnxruntime with the TIM-VX API by this VSINPU execution provider.
### Description
Update upstream packaging pipeline name to make it more meaningful.
### Motivation and Context
The upstream pipeline used to only building Nuget packages, but now it
also builds Zip and Java. So change the name will make it more
meaningful.
### Description
1. Update the functions in tensorprotoutils.h to use
std::filesystem::path instead of onnxruntime::Path. Eventually we can
remove the whole onnxruntime::Path class, but to this PR small I am not
doing that.
2. Remove the _SILENCE_EXPERIMENTAL_FILESYSTEM_DEPRECATION_WARNING macro
def when TensorRT EP is enabled.
### Description
Vitis AI EP synchronously supports the TensorProto data types supported
by ONNX 1.16.
Add error message show when graph resolve fail for troubleshooting.
### Motivation and Context
ONNX 1.15 & 1.16 add support some new TensorProto DataType , such as
- FLOAT8E4M3FN
- FLOAT8E4M3FNUZ
- FLOAT8E5M2
- FLOAT8E5M2FNUZ
- UINT4
- INT4
---------
Co-authored-by: liumingyue <mingyue@xilinx.com>
### Description
Provide user level options to control the fallback on CPU for models not
supported on Intel's NPU hardware.
### Motivation and Context
- Current workflow of OVEP allows safe fallback from OV NPU to OV CPU on
compilation failures. Also supports MLAS CPU fallback in presence of
unsupported custom ops.
- The PR provides a build-time option to disable fallback from OV NPU to
OV CPU.
- The session Option "kOrtSessionOptionsDisableCPUEPFallback" disables
OV CPU and MLAS CPU fallback.
- Also has bug fix for proto creation.
---------
Co-authored-by: jatinwadhwa921 <jatin.wadhwa@intel.com>
Co-authored-by: ankitm3k <ankit.maheshkar@intel.com>
### Description
1. Add QNN UTs for QNN Pad Op with FP16 data on HTP backend
2. Improve Pad op builder to handle invalid optional input
3. Add UT for ReduceSum for FP16 precision with 5D for issue reproduce
### Description
<!-- Describe your changes. -->
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
As suggested by SciPy's doc, we will
`Build against NumPy 2.0.0, then it will work for all NumPy versions
with the same major version number (NumPy does maintain backwards ABI
compatibility), and as far back as NumPy 1.19 series at the time of
writing`
I think it works because in
[numpyconfig.h#L64](https://github.com/numpy/numpy/blob/main/numpy/_core/include/numpy/numpyconfig.h#L64)
there is a macro NPY_FEATURE_VERSION. By default it is set to
NPY_1_19_API_VERSION. And the NPY_FEATURE_VERSION macro controls ABI.
This PR only upgrade the build time dependency; When a user installs
ONNX Runtime, they still can use numpy 1.x.
### Motivation and Context
Recently numpy published a new version, 2.0.0, which is incompatible with the latest ONNX Runtime release.
### Description
### Motivation and Context
The pipeline is green even Llama2 parity_check fails.
The PR should be merged after the below exception is solved.
'''
2024-06-25 03:49:43.621298481 [E:onnxruntime:,
sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned
while running Expand node. Name:'/model/Expand' Status Message:
/model/Expand: left operand cannot broadcast on dim 3 LeftShape:
{1,1,9,9}, RightShape: {2,1,9,17}
An error occurred while verifying parity: Error in execution: Non-zero
status code returned while running Expand node. Name:'/model/Expand'
Status Message: /model/Expand: left operand cannot broadcast on dim 3
LeftShape: {1,1,9,9}, RightShape: {2,1,9,17}
Traceback (most recent call last):
File
"/workspace/onnxruntime/python/tools/transformers/models/llama/convert_to_onnx.py",
line 1043, in main
parity_check(parity_cmd)
File
"/workspace/onnxruntime/python/tools/transformers/models/llama/llama_parity.py",
line 298, in main
verify_parity(args, location, use_auth_token, kv_cache_ortvalues,
pytorch_model=llama, config=config)
File
"/workspace/onnxruntime/python/tools/transformers/models/llama/llama_parity.py",
line 137, in verify_parity
ort_model.run_with_iobinding(io_binding)
File
"/home/onnxruntimedev/.local/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py",
line 331, in run_with_iobinding
self._sess.run_with_iobinding(iobinding._iobinding, run_options)
RuntimeError: Error in execution: Non-zero status code returned while
running Expand node. Name:'/model/Expand' Status Message: /model/Expand:
left operand cannot broadcast on dim 3 LeftShape: {1,1,9,9}, RightShape:
{2,1,9,17}
'''
The exception looks caused by #19832
ONNX's Expand supports bidirectionally broadcast, while WebNN's expand
op only supports unidirectionally broadcast. Thus we should calculate
the output shape for 'newShape' input of WebNN's expand op.
### Description
<!-- Describe your changes. -->
The split op is using pin_memory when split on different sizes.
But pin_memory is not capable for using cudagraph.
Add a new implementation for only transformer scenarios, it split the
qkv_proj into q, k, v, not using pin_memory.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
<!-- Describe your changes. -->
Setting the log level after environment creation is too late in some
cases.
If the DML EP is enabled, it will create a composite sink with the
original logger using the creation time log severity, as well as
additional ETW sink. As it saves the current severity levels for each
sink inside the composite sink that prevents being able to get verbose
log output to stdout even if you set that at the session level.
I don't know enough about the setup that combines ETW with the original
sink to say whether we should also be updating the severity of
individual sinks in the combined sink, so this change is limited to
making the unit tests behave in the expected manner when the default log
severity is set in the background and not directly controlled.
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Make it possible to get verbose output to stdout when the DML EP is
enabled.
### Description
Skip softmax BF16 test for ROCm, because BFloat16 is unsupported by
MIOpen, and `torch.cuda.is_available()` also returns `True` for ROCm.
### Problem
Newer models using more novel equations (e.g. `bhwc,hkc->bhwk` in
Segment Anything's encoder or `bqc,bchw->bqhw`) cause fallback from DML
to CPU, yielding performance issues. The EP had some pattern matching to
map more common equations to existing DML operators, but the number of
permutations was prohibitive and could not catch them all.
### Solution
So, ditch the static mapping, and instead handle any 1-input or 2-input
cases via remapped strides and a mini-graph of elementwise
multiplication & sum reduction (as if DML had a
`DML_OPERATOR_DOT_PRODUCT` that took `axes`). A subset of mappings still
exist for performance (GEMM, pure reduction, transpose...), but they are
identified generally rather than via a pattern table. Also...
- Diagonals are supported now (e.g. iji->i).
- Removes any remaining DML-specific EinSum `GTEST_SKIP` statements.
- Handles any cases up to 8 unique labels (DML dimension limit is 8D).
- \>= 3 inputs and arbitrary size inputs via ellipsis are not handled,
but we have yet to come across a model.
### Description
Alternative design from #20942
Allow users to pass in a model path for the generate_artifacts API.
### Motivation and Context
- ONNX API calls such as the onnx checker + shape inference fail when
given a model > 2GB, but work if a path to a model >2GB is passed in.
Context and motivation:
When quantizing large transformer models, we faced OOM issue when the
number of calibration samples goes up. To resolve this, in the PR we
want to add support for reading quantization data in chunck, calculating
ranges for intermediate tensors, then accumulating results for the final
ranges.
### Description
This reverts commit 1d7bf56947 because it
broken the AMD GPU CI pipeline. Sorry when I reviewed the PR I forgot to
run the AMD GPU CI pipeline.
Will revert the PR first then ask the author to fix the issue.
### Description
Update protobuf_cmake.patch to allow extra disablements. ORT repo
already patches protobuf to not disable the warning 4996.
### Motivation and Context
To meet SDL requirements, Microsoft repos have to fail build if there is
warning 4996
Binskim also gives errors if warning 4996 is disabled.
We can suppress the Binskim issues, but we need a way to disable the
warnings for the minimal set of code that has them.
Right now, WindowsAI disables 4996 for entirety of ORT, but it should
only be disabled for protobuf.
### Description
<!-- Describe your changes. -->
### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
### Description
Under certain conditions with enabling & disabling ETW continuously, we
got a crash report.
Allows ETW callbacks to be de-registered upon class destructor.
Related to #20537
### Motivation and Context
Fixes crash
### Callstack
We see it crash in
[0x0]
onnxruntime!<lambda_967a738fca8512372f170fcaf2d094d4>::operator()+0x34
0x12941ff570 0x7ffa994f0a04
[0x1] onnxruntime!std::_Func_class<void,_GUID const *,unsigned
long,unsigned char,unsigned __int64,unsigned
__int64,_EVENT_FILTER_DESCRIPTOR *,void *>::operator()+0x54 0x12941ff7b0
0x7ffa994f0d64
[0x2]
onnxruntime!onnxruntime::logging::EtwRegistrationManager::InvokeCallbacks+0xcc
0x12941ff7b0 0x7ffa994f0d64
[0x3]
onnxruntime!onnxruntime::logging::EtwRegistrationManager::ORT_TL_EtwEnableCallback+0x94
0x12941ff860 0x7ffa98d19628
and seems to us that the this pointer captured in
etwRegistrationManager.RegisterInternalCallback(
[&etwRegistrationManager, this](
...
is no longer valid when the callback is called.
### Description
The machine has multiple python installations and none of them is in
PATH. Therefore we should explicitly set python version via this task to
avoid having surprises.
### Motivation and Context
Similar to #21095
### Description
Delete RoslynAnalyzers. Use CodeQL instead.
### Motivation and Context
Now we already have CodeQL which is modern and also covers C# code. The
RoslynAnalyzers one is not in our pull request pipelines. The
"RoslynAnalyzers@2" task is outdated and needs be upgraded. I will
delete it for now since we already have CodeQL.
### Description
1. added kernel to quantize matmul B tensor to q4, and store in the same
shape as original tensor. scales and zero points are calculated as well.
scales and zero points have the same shape.
2. added kernel to transpose q4 B tensor to B tensor in MatMulNBits.
Scales and zero points are transposed as well.
####
Benchmark
<1024 x 4096 input, 64 quant block, 8 threads>:
- quantize: 23035923 ns
- transpose: 718635 ns
<1024 x 4095 input, 64 quant block, 8 threads>:
- quantize: 26759319 ns
- transpose: 1279064 ns
### Motivation and Context
The MatMulNbits tool chain current only supports converting a MatMul op
direct to MatMulNBits op. MatMulNbits op is not an ONNX standard op.
Therefore, we need the tool chain to support converting MatMul to Q/DQ
format, and later in the transform step converts DQ + MatMul to
MatMulNBits. The tensors stored in DQ are the quantized constants and
will be stored in the MatMulNBits.
### Description
change is_pod tp is_trivial
### Motivation and Context
This is commonnly needed for both linux and win c++20 upgrade.
is_trivial was introduced backed in C++11
### Description
Remove the "--enable_language_interop_ops" build flag, because the code
is incompatible with the latest numpy, and the build flag is not used
anywhere except a macOS CI pipeline. It does not seem to have a ship
plan.
### Motivation and Context
The build error was:
```
onnxruntime/core/language_interop_ops/pyop/pyop.cc:122:85: error: no member named 'elsize' in '_PyArray_Descr'
static_cast<int64_t>(PyArray_DescrFromType(type)->elsize),
~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
```
