Commit graph

1281 commits

Author SHA1 Message Date
Adrian Lizarraga
f20e475e67
[QNN EP] Update QNN SDK to version 2.14.1 (#17467)
### Description
Updates the version of QNN SDK used by CI Pipelines. Enables some tests
fixed by 2.14.1, but still need to look into Resize in a separate PR.

### Motivation and Context
Test latest version of QNN SDK.
2023-09-11 21:07:50 -07:00
Yulong Wang
850baced33
[web] a few updates to web pipeline (#17485)
### Description

Update the Web CI pipelines:

- remove parameter 'WebTemplate': Since we start to support webgpu, the
linux-web-ci.yml is no longer working and it is already out-of-date.
remove this file and parameter so that we always use win-web-ci.yml

- change flag `RunWebGpuTests` into 2 flags, for release and debug.
Currently for CI we only run webgpu tests on release build. But we want
to have the capability to run webgpu tests on debug build as well.


After this PR is merged, next step is to enable both Debug and Release
webgpu tests in PostMerge pipeline.
2023-09-11 11:43:42 -07:00
Caroline Zhu
dcc93909b4
Add training WASM generation to Web CI pipeline (#17319)
### Description
[Successful pipeline
run](https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=1123141&view=results)

Added flag to build the training artifacts & updated the
pull-wasm-artifacts script to pull the training artifacts as well.

Bundled into this PR are minor formatting fixes + naming fixes.

### Motivation and Context
[This PR](https://github.com/microsoft/onnxruntime/pull/16521) extended
the WASM API wrapper to build training WASM artifacts as well.
The ORT training WASM artifacts are required to support ORT training web
bindings.
2023-09-08 15:49:47 -07:00
Changming Sun
bc84f52633
Update C/C++ dependencies: abseil, date, nsync, googletest, wil, mp11, cpuinfo and safeint (#15470)
### Description
Update C/C++ dependencies abseil, date, nsync, googletest, wil, mp11,
cpuinfo and safeint to newer versions per request of @
mayeut. He created the following PRs to update the deps:
https://github.com/microsoft/onnxruntime/pull/15432
https://github.com/microsoft/onnxruntime/pull/15434
https://github.com/microsoft/onnxruntime/pull/15435
https://github.com/microsoft/onnxruntime/pull/15436
https://github.com/microsoft/onnxruntime/pull/15437

However, our build system needs to fetch the dependencies from an
internal mirror that only Microsoft employees have write access to. So I
closed his PRs and created this one.

This PR also updates abseil to a newer version. This is to prepare for
upgrading re2.
2023-09-08 13:35:04 -07:00
Ashwini Khade
c5dbd5c919
Updates to training pipelines (#17292) 2023-09-08 11:57:12 -07:00
Yi Zhang
ae74a517b6
Run Nuget_Test_Linux_GPU in container (#17452)
### Description
<!-- Describe your changes. -->



### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->

### Verification

https://dev.azure.com/aiinfra/Lotus/_build/results?buildId=351542&view=results
2023-09-08 13:41:20 +08:00
Yi Zhang
0a3eb60b01
Fix Bug: Step failed but not exited with error (#17442)
### Description
Add "set -ex" in the script.


### Motivation and Context
Build failed but it still passed.

https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=1132003&view=logs&j=7536d2cd-87d4-54fe-4891-bfbbf2741d83&t=39e3f98f-7fe5-578c-20bd-5ae5a4590bda
2023-09-07 14:33:31 +08:00
Changming Sun
b38fb0da06
Revert the yaml file changes in "Nodejs_Packaging_CPU" build job (#17441)
### Description
The yaml file changes made in #16050 do not really work. Currently the
pipeline is failing with error:
```
Error: Not found SourceFolder: C:\a\_work\5\b\RelWithDebInfo\RelWithDebInfo\nuget-artifacts\onnxruntime-win-x64\lib
```

So, I will revert the yaml changes first to bring the pipeline back.
Some people are waiting for our nightly packages.

Test run:
https://aiinfra.visualstudio.com/Lotus/_build/results?buildId=351104&view=results

### Motivation and Context
2023-09-06 20:20:55 -07:00
Yi Zhang
ede339f304
Move dotnet build and test into docker in Linux CPU CI (#17417)
### Description
install dotnet 6.0 in the docker image.
move C# build and test into docker.

### Motivation and Context

### Note
The Unit tests and Symbolic shape infer's migration will be in another
PR.
2023-09-07 09:28:16 +08:00
Edward Chen
a3a1237270
Disable xcpretty filtering of xcodebuild output in iOS packaging pipeline. (#17429) 2023-09-06 09:04:17 -07:00
Changming Sun
c6b0d185b4
Update cmake to 3.27 and upgrade Linux CUDA docker files from CentOS7 to UBI8 (#16856)
### Description
1. Update docker files and their build instructions.
ARM64 and x86_64 can use the same docker file.

2. Upgrade Linux CUDA pipeline's base docker image from CentOS7 to UBI8
AB#18990
2023-09-05 18:12:10 -07:00
Rachel Guo
b54619509f
Refine build script for adding disable selected data types option (#17284)
### Description
<!-- Describe your changes. -->

As title. 

### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->

Now we have multiple data types that we want to disable for minimal
build and to reduce binary size. may be worth adding an argument in the
build script for specifying that.

Also for fp16 type stuff, it may be too restrict to disable that for all
minimal build.

---------

Co-authored-by: rachguo <rachguo@rachguos-Mac-mini.local>
2023-08-31 13:32:55 -07:00
Yi Zhang
507a40e1e9
Add compiler cache in Linux GPU TensorRT CI. (#17348)
### Description
Add the compiler cache in linux GPU tensorRT CI.
Save about 30 minutes in the GPU machine. (52 minutes -> 24 minutes)

PS. 
There're only white-space differences in the dockerfile.

### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
2023-08-31 08:13:26 +08:00
Changming Sun
71da0824f3
Upgrade binskim and fix an error in nuget packaging pipeline (#17340)
### Description
Upgrade binskim and fix an error in nuget packaging pipeline.
2023-08-30 07:52:06 -07:00
Jian Chen
922629aad8
Upgrade Centos7 to Alamlinux8 (#16907)
### Description
<!-- Describe your changes. -->



### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Get the latest gcc 12 by default

---------

Co-authored-by: Changming Sun <chasun@microsoft.com>
2023-08-29 21:05:36 -07:00
Yi Zhang
d4a61ac71f
Pr trggiers generated by code (#17247)
### Description
1. Refactor the trigger rules generation.
2. Skip all doc changes in PR pipelines.


### Motivation and Context 
Make all trigger rules generated by running set-trigger-rules.py to
reduce inconsistences.
It's easily to make mistakes to copy&paste manually. 

For example: these 2 excludes are different, Why?

4e6cec4d09/tools/ci_build/github/azure-pipelines/linux-ci-pipeline.yml (L16-L18)


4e6cec4d09/tools/ci_build/github/azure-pipelines/linux-gpu-ci-pipeline.yml (L27-L29)


### Note
All changes in workflow yamls are generated by code.
Please review the **skip-js.yml, skip-docs.yml and
set-trigger-rules.py**.

@fs-eire, please double check the 
filter rules in skip-js.yml
and the skipped workflows

7023c2edff/tools/ci_build/set-trigger-rules.py (L14-L41)
2023-08-30 05:57:03 +08:00
Yi Zhang
0e9e9b2a67
Fix one exception in post merge (#17327)
### Description
<!-- Describe your changes. -->



### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
2023-08-29 19:24:50 +08:00
cloudhan
bf8b1681f9
Build nuget pkg for ROCm (#16791)
Add nuget pkg building and publishing for ROCm EP

---------
Co-authored-by: Yi Zhang <zhanyi@microsoft.com>
2023-08-28 13:35:08 +08:00
Yifan Li
808215366d
Fix Multi GPU TensorRT tests (#17269)
### Description
* Integrate `trt_multi_gpu` test stage in ORT post merge CI (Win-2xA10
vm)
* Deprecate Linux MultiGPU TRT CI (This vm will be deprecated soon)
* Add multi gpu support to existing C# test cases
* Deprecate unfunctional flag `--enable_multi_device_tests`

### Motivation and Context
* Two contexts of replacing Linux MultiGPU TRT CI:
* Flag `--enable_multi_device_tests` is not functional, which cannot
detect issues like #17036
* The Linux-2xM60 VM of this CI pool is about to be deprecated 9/6/23.
Need to enable this test in other dualGPU vm pool.
2023-08-25 20:30:45 -07:00
Arthur Islamov
c262879214
Added DML and CUDA provider support in onnxruntime-node (#16050)
### Description
I've added changes to support CUDA and DML (only on Windows, on other
platforms it will throw an error)



### Motivation and Context
It fixes this feature request
https://github.com/microsoft/onnxruntime/issues/14127 which is tracked
here https://github.com/microsoft/onnxruntime/issues/14529

I was working on StableDiffusion implementation for node.js and it is
very slow on CPU, so GPU support is essential.

Here is a working demo with a patched and precompiled version
https://github.com/dakenf/stable-diffusion-nodejs

---------
2023-08-25 16:57:06 -07:00
Yi Zhang
9cd33e07b4
Readd Tests in Window GPU Reduced Ops workflow (#17294)
### Description
Add single test step in Window GPU Reduced Ops workflow


### Motivation and Context
The old workflow's building and testing were running in one command.
In PR #17263, the test step was removed by mistake.
So, readd it.
How to consolidate the test step is in consideration.
2023-08-25 15:56:59 +08:00
Yi Zhang
756eda2cc4
Windows CI build steps template (#17263)
### Description
1. New windows ci build steps template.
2. Remove useless variables.

### Motivation and Context
1. Make it easier to apply build cache to all windows CIs.
2. Other team's devs only need to take care of build options


###Comparision
Before: 

9f21f694cf/tools/ci_build/github/azure-pipelines/win-gpu-tensorrt-ci-pipeline.yml (L19-L82)

After:
b4c1f2261b/tools/ci_build/github/azure-pipelines/win-gpu-tensorrt-ci-pipeline.yml (L35-L54)
2023-08-25 05:58:49 +08:00
Jian Chen
33415b9da4
Removing 10.14 suffix from osx nuget package (#17277)
### Description
<!-- Describe your changes. -->



### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
2023-08-24 08:51:54 -07:00
cloudhan
87bef1f3f2
Move composable_kernel to deps.txt (#17245) 2023-08-23 17:39:16 -07:00
Yi Zhang
61a79436e2
Common pre-build steps of Windows CI (#16970)
### Description
Unify some pre-build common steps.

### Motivation and Context
In the long run, other devs should only focus on build option and test
commands.
It would reduce mistakes and maintenance cost to use common template
steps.
There will be more PRs to achieve the goal.
2023-08-22 18:09:55 +08:00
cloudhan
4e6cec4d09
Update ck and enable test (#16383)
Apply the fix in https://github.com/ROCmSoftwarePlatform/composable_kernel/issues/728
Introduce more kernel instances and allow the introduction of streamk and splitk.
2023-08-22 11:08:55 +08:00
Baiju Meswani
aae9a52e8b
Avoid pushing cpu package to https://download.onnxruntime.ai/ (#17238) 2023-08-21 15:47:07 -07:00
Chi Lo
9445539e2c
Update dependency for deps.txt (#17220)
https://github.com/microsoft/onnxruntime/pull/17059 updates deps.txt and
we also need to update cgmanifest.json and upload the files to Azure
DevOps


https://aiinfra.visualstudio.com/Lotus/_build/results?buildId=342803&view=results
for testing
2023-08-19 00:43:25 -07:00
Yulong Wang
3426954525
disable browser stack tests (#17224)
### Description
disable browser stack tests
2023-08-18 17:14:12 -07:00
Changming Sun
6db72165eb
Fix python packaging test pipeline (#17204)
### Description
1. Fix python packaging test pipeline. There was an error in
tools/ci_build/github/linux/run_python_tests.sh that it installed a
released version of onnxruntime python package from pypi.org to run the
test. Supposedly it should pick one from the current build.
2. Refactor the pipeline to allow choosing cmake build type from the web
UI when manually trigger a build. Now this feature is for Linux only.
Because I don't want to change too much when we are about to cut a
release branch. After that I will expand it to all platforms. This
feature is useful for debugging pipeline issues, also, we may consider
having a nightly pipeline to run all tests in Debug mode which may catch
extra bugs because in debug mode we can enforce range check.

Test run:
https://aiinfra.visualstudio.com/Lotus/_build/results?buildId=342674&view=results

### Motivation and Context
Currently the pipeline has a crash error. 

AB#18580
2023-08-18 14:51:26 -07:00
Adrian Lizarraga
6ee4be724b
Update LICENSE name in NuGet packaging pipelines (#17183)
### Description
Updates NuGet packaging pipelines to use the correct license name.

### Motivation and Context
The license name changed. See https://github.com/microsoft/onnxruntime/pull/17170
The QNN_Windows_Nuget and Zip-Nuget-* pipelines will not run without this update.
2023-08-17 22:22:19 -07:00
Changming Sun
0cccbcc47b
Move DML build job's Prefast task to a CPU machine pool (#17192)
### Description
Move DML build job's Prefast task to a CPU machine pool which has larger
memory. The current one runs out of memory in every run.

### Motivation and Context
To fix the broken python packaging pipeline.
2023-08-17 13:16:29 -07:00
Jian Chen
e0022d061f
Set web-ci-pipeline.yml only triggered when related fields are updated (#17148)
- 'js/web'
    - 'js/node'
    - 'onnxruntime/core/providers/js'
    is updated

### Description
<!-- Describe your changes. -->



### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
2023-08-17 12:55:35 -07:00
Adrian Lizarraga
96b1ff610b
Add CI and PR validation triggers to QNN Windows x64 Pipeline yaml (#17178)
### Description
Adds continuous integration and pull-requestion validation triggers
directly to the yaml file for the Windows x64 QNN CI Pipeline.


### Motivation and Context
There have been various unit tests failures that break the
QNN_Windows_Nuget pipeline, which builds QNN EP for Windows x64. This PR
ensures that QNN EP is built and tested on a Windows x64 image for every
pull request.
2023-08-16 11:44:54 -07:00
Jian Chen
8998b6811d
Fix NPM Packaging Pipeline (#17182)
### Description
<!-- Describe your changes. -->



### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
2023-08-15 22:56:38 -07:00
Adam Louly
c647e3e8ab
Run nightly pipeline tests from the commit id. (#17162)
### Description

The onnxruntime-CI-nightly-ort-pipeline encounters occasional failures
due to synchronization discrepancies between the ACPT nightly image and
the repository. We are addressing this by executing tests using the
commit ID associated with the ort build within the ACPT image.

---------

Co-authored-by: Adam Louly <adamlouly@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>
2023-08-15 12:07:38 -07:00
Changming Sun
8e203efc69
Cleanup cmake file (#17154)
### Description
1. Clean up cmake files. Remove some unused code
2. Remove the "Semmle" task from
tools/ci_build/github/azure-pipelines/templates/win-ci.yml. Semmle is
deprecated and replaced by CodeQL.
2023-08-15 10:51:33 -07:00
Changming Sun
2a22325005
Explicitly set JDK version when building ORT java package (#17147)
### Description
Explicitly set JDK version when building ORT java package. This is to fix an internal build error.
2023-08-15 10:36:05 -07:00
Adrian Lizarraga
b734db1924
[QNN EP] Fix CI build on Windows x64 pipelines (#17152)
### Description
- Disables Resize tests that use nearest mode on QNN CPU.
- Fixes indentation problems on yaml for win x64 qnn pipeline.


### Motivation and Context
The QNN windows Nuget pipeline does not run due to failing unit tests on
Windows x64. These tests should not be enabled until we determine the
rounding behavior of QNN's ResizeNearestNeighbor operator.
2023-08-14 21:03:14 -07:00
Baiju Meswani
289600b47d
ONNX Runtime training cpu package name for ADO (#17109) 2023-08-14 11:32:35 -07:00
Jian Chen
45f52987a2
Web CI Pipeline Isolation (#17005)
### Description
<!-- Describe your changes. -->



### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
2023-08-14 10:37:37 -07:00
Jian Chen
68ea9631af
Fix typo onnxruntimecpubuilpython (#17120)
### Description
The correct name should be  onnxruntimecpubuildpython



### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->

Co-authored-by: Yulong Wang <7679871+fs-eire@users.noreply.github.com>
2023-08-14 08:34:43 -07:00
Changming Sun
4728f20f9a
Fix CI build (#17118)
### Description
Some pipelines are failing. It is because PR #16325 set ONNX version to
`rel-1.14.1` . It is a branch name, not a commit or tag name. It means
whenever the branch got a new commit, we will auto pick it and use it.
2023-08-11 10:56:38 -07:00
Edward Chen
e7e974b23f
Use double quotes so variable gets expanded. (#17105) 2023-08-11 09:05:41 -07:00
Hector Li
344c41fdb9
[QNN EP] Update QNN to v2.13 (#17079)
### Description
Update QNN SDK to v2.13, update some UTs accordingly
2023-08-10 20:47:55 -07:00
Yulong Wang
9cd4e5af68
[wasm] upgrade emsdk to 3.1.44 (#17069)
### Description
This change upgrade emsdk to 3.1.44.

Because backend is upgraded to LLVM 16, so need to fix a lot of build
failures caused by "-Wshorten-64-to-32".

most of the build failures comes from generated `onnx.pb.h`, and this
can be fixed by including "core/graph/onnx_protobuf.h", which detects
and ignore shorten-64-to-32 warnings.
2023-08-10 16:08:36 -07:00
Bowen Bao
6986981482
Bump ONNX version (#16325)
### Description
Bump ONNX version to https://github.com/onnx/onnx/tree/rel-1.14.1 to
include a fix for segfault when shape inferencing nested onnx functions.



### Motivation and Context
Resolves #16170
2023-08-10 11:27:28 -07:00
PeixuanZuo
12837ba5c7
[ROCm] Update CI based on ubuntu 22.04 (#17076)
- Update ROCm version to ROCm5.6
- Update CI based on ubuntu 22.04
2023-08-10 09:51:29 -07:00
RandySheriffH
a7542f48d6
Make AzureEP default for python and c# packaging (#17025)
Make AzureEP default for python and c# packaging, with UT.

---------

Co-authored-by: Randy Shuai <rashuai@microsoft.com>
2023-08-09 12:36:52 -07:00
Yulong Wang
56bced0581
[js/web] enable webgpu in browser unit test (#16310)
### Description
enable webgpu in browser unit test.

The CI pipeline uses Edge v113+ which enables WebGPU.

===

**UPDATE on 08/07/2023:**
- add flags to Edge browser launch commandline so that Edge on CI agents
can initialize WebGPU correctly.
- ONLY enable webgpu on web release build. Other pipelines are using
flag `-b=wasm,webgl,xnnpack` to specify the other 3 backends explicitly.
- disable "Resize" related test failures. Once they are fixed the tests
can be re-enabled.

---------

Co-authored-by: Satya Jandhyala <satya.k.jandhyala@gmail.com>
2023-08-08 11:45:04 -07:00