Commit graph

258 commits

Author SHA1 Message Date
Scott McKay
fb4a8e12fc
Limit inclusion of Xamarin mobile target frameworks. (#9834)
- Only set them as targets for the ORT nuget package
    - Use OrtPackageId as the condition for inclusion, if installed
      - need to do the nuget restore via msbuild so that this property is set correctly
  - Add desktop-only version of the C# sln as there is no way to exclude the mobile specific csproj's from an sln
    - use this when applicable if someone is running build.py with the `--build_nuget` flag

Other
  - remove attempt to include symbols in the nuget package as nuget doesn't support symbols in native packages
  - update build.py to use `nuget` and not a windows specific path and filename for a linux build with `--build_nuget`
2021-11-23 11:29:53 +10:00
Changming Sun
88d8d2d02a
Remove Copy NuGet Package to Azure Blob Store build step (#8991) 2021-11-22 08:03:02 -08:00
Chi Lo
7242627fec
Integrate TensorRT into GPU Python package (#9785)
* add use_tensorrt build option

* Add use_tensorrt to running tests

* add use_tensorrt for Windows

* make trt ep to skip backend test

* make trt ep to skip backend test

* Fix bug

* Add/Modify description

* modify for debug

* swtich pool to test

* modify to debug

* modify to debug

* add vobersity

* refine the code

* refine the code

* refine the code

* fix flake8 warning

* refine the code

* add pre_load check for trt as well as add cupti lib to cuda depedencies

* modify script to make trt build path the same as cuda

* show error message when user wants to run TensorRT but TensorRT is not installed in the env

* fix bug

* fix bug

* add trt lib for manylinux

* include cuda_dependencies for trt

* rewrite the condition to throw exception

* make code more compact
2021-11-18 13:26:51 -08:00
Chun-Wei Chen
ac57afc3a6
Update ONNX to 1.10 globally in CIs (#9751)
* Bump ONNX 1.10.2 globally

* load ONNX_VERSION from VERSION_NUMBER

* /
2021-11-15 15:28:26 -08:00
Yulong Wang
31dc768e07
update ONNX Runtime Web CI to use same script for package versioning (#9698)
* use ps script for package generating

* allow e2e test has no common pkg
2021-11-10 12:52:34 -08:00
Changming Sun
53afaefe3b
Refactor Windows CI pipeline yaml files (#9672) 2021-11-08 11:11:49 -08:00
Yulong Wang
c6fddb263f
Add Node.js binding support to packaging pipeline (#9577) 2021-11-05 15:29:40 -07:00
Changming Sun
74dc48f61c
Remove python 3.6 from macos pipeline (#9602) 2021-11-04 19:36:44 -07:00
Guoyu Wang
9cedb12c89
Add App Center e2e test to Android package (#9653)
* Add app center test for Android pipeline

* update android test path

* Add logging of system abi

* fix java spotless error
2021-11-03 22:56:46 -07:00
Changming Sun
2406a425a7
Move the common part of Windows CPU CI pipeline to a template file (#9650) 2021-11-03 17:35:51 -07:00
liqun Fu
15867dcc08
enable prefast check with orttraining win pipeline (#9638)
* enable prefast check with orttraining win pipeline

* timeout to 200

Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
2021-11-03 15:20:44 -07:00
Changming Sun
cc73bcc243 Suppress component governance component warnings for ios 2021-10-28 14:25:29 -07:00
Changming Sun
87b1fddd97
Add Linux/MacOS ARM64 support to nuget packaging pipeline (#9570) 2021-10-27 19:00:43 -07:00
Changming Sun
aa76520e60
Update macOS build agents to macOS 11 (#9562) 2021-10-27 10:00:04 -07:00
Scott McKay
b5a652c578
Add Xamarin support (#9436)
Add Xamarin support to the ORT nuget packages.
  - Update C# code to support Xamarin builds for iOS and Android
  - refactor some things to split out common code
  - include iOS and Android ORT native shared library in native nuget package
2021-10-27 20:07:07 +10:00
Yulong Wang
bf4c3fa3d6
[node.js binding] aggregate binaries for multiple platforms in single NPM package (#9501) 2021-10-25 20:16:10 -07:00
Changming Sun
334980e016 Delete nocontribops pipelines 2021-10-06 10:30:32 -07:00
Sheil Kumar
8f6fd014e4
Force Windows AI NuGet pipeline to use Windows SDK 19041 (#9255)
* Force Windows AI Nuget pipeline to use 19041 Windows SDK as 22000 casues a downlevel regression by importing LoadLibraryW

* move into quotes

Co-authored-by: Sheil Kumar <sheilk@microsoft.com>
2021-10-01 21:46:14 -07:00
Yulong Wang
448325b254
[js/web] name ort web for consistency (#9240) 2021-09-30 22:53:26 -07:00
Yulong Wang
8c57d51928
support WebAssembly SIMD for qgemm (#9191)
* support WebAssembly SIMD for qgemm

* remove '--experimental-wasm-bulk-memory' for test
2021-09-30 12:40:56 -07:00
Sheil Kumar
f16bb37fae
make uwp store apps link to statically-linked crt desktop builds (#9182)
Co-authored-by: Sheil Kumar <sheilk@microsoft.com>
2021-09-27 15:37:15 -07:00
Suffian Khan
47888392ab
Fix nightly CI pipeline to generate ROCm 4.2 wheels and add ROCm 4.3.1 wheels (#9101)
* make work for both rocm 4.2 and rocm 4.3.1

* fix rocm 4.3.1 docker image reference

* fix CUDA_VERSION to ROCM_VERSION

* fix ReduceConsts conflict def

* add ifdef to miopen_common.h as well

* trailing ws
2021-09-19 23:36:03 -07:00
Yulong Wang
0183492931
simplify Web CI pipeline (#9094) 2021-09-17 13:00:07 -07:00
Guoyu Wang
438175cb34
Build shared host protoc in iOS CI pipelines (#9087)
* iOS build, share host protoc build

* Change android ci

* ios packaging ci

* checkout submodule

* revert

* update package pipeline

* minor update
2021-09-16 17:16:50 -07:00
Yulong Wang
498461485f
optimize WebAssembly CI pipeline (#9058) 2021-09-16 01:07:36 -07:00
Guoyu Wang
bee5c26580
Add CPU_ONLY runtime option to NNAPI EP (#9066)
* Add NNAPI cpu only option

* update java

* Update comments
2021-09-15 15:50:18 -07:00
Suffian Khan
4322f7e647
Fix ROCm wheels CI pipeline break by installing latest protobuf from source (#9047)
* install protobuf from source

* fix rm command in Dockerfile

* fix options on rm command

* fix cd into protobuf source directory

* try again

* remove strip step

* debug list the files

* ls on /usr

* more debug

* more debug

* adjust LD_LIBRARY_PATH

* try remove protobuf before ORT build
2021-09-14 12:07:00 -07:00
Guoyu Wang
cf70635d2a
Add Android executable drop in the Package pipeline (#9050)
* add copy executable for android job

* minor fix

* Variable fix

* Move to use tgz because zip is not part of the docker image

* update compression
2021-09-14 11:45:33 -07:00
Guoyu Wang
c709380c52
Add full iOS job in package pipeline (#9036)
* Add full ios xcframework job

* create zip file of the xcframework
2021-09-13 15:54:11 -07:00
Edward Chen
29d6573f3d
Increase timeouts for Mac CI builds. (#9024)
Increase timeouts for "orttraining-mac-ci-pipeline" and "iOS CI Pipeline" CI builds.
2021-09-10 12:57:08 -07:00
Guoyu Wang
1533f574e4
Add full Android job in package pipeline (#9009)
* Add full Android job in package pipeline

* Address CR comments
2021-09-08 21:12:59 -07:00
Chi Lo
1f576e1766
Detect necessary files inside GPU packages (#8955)
* Rename files

* Update YAML files

* Update validation script and YAML
2021-09-03 13:28:28 -07:00
Changming Sun
fbb6f0f599 Fix an error in Nuget pipeline caused by merge conflict 2021-09-02 09:26:25 -07:00
liqun Fu
f126a12699
decouple pytorch from onnxruntime training build (#8815) 2021-09-01 16:31:53 -07:00
Changming Sun
6299a60bf8
Nuget: splitting PDB files to a separated package (#8903) 2021-09-01 09:07:24 -07:00
Changming Sun
a9a0d3f6fa Update min supported macOS version to 10.14 2021-08-31 16:09:48 -07:00
Changming Sun
c6d9426ef2
Add binary size reporting back (#8883) 2021-08-30 19:48:38 -07:00
Guoyu Wang
6a1939252f
Fix Android java API failure (#8865)
* Fix Android Package break

* Without java fix -- pipeline should fail

* With java fix, should pass now

* address CR comments
2021-08-27 15:58:56 -07:00
Chi Lo
6a477acecf
Add tensorrt_provider_factory.h to artifact (#8869) 2021-08-27 09:09:54 -07:00
Chi Lo
32ecbf4691
Create combined GPU tarball and zip file package (#8827)
* Add onnxruntime_providers_shared.dll into gpu nuget package

* Modify for test

* Temporarily remove for test

* Modify for test

* Modify for test

* Test packging Windows combined GPU

* Test packging Windows combined GPU

* Test packging Windows combined GPU

* Test packging Windows combined GPU

* modify for test

* modify for test

* fix bug

* Modify for test

* Modify for test

* Modify for test

* Modify for test

* Modify for test

* Modify for test

* Modify for test

* Modify for test

* Prepare for PR

* Prepare for PR

* Code refactor

* Rename proper Artifact name

* Rename intermediate Artifact names

* Revert Artifact Names

* Rename Artifact Names

* Modify Artifact name

* Modify Artifact name

* Modify Artifact name

* Update Java package

* Update Java package

* fix bug to change artifact name

* Fix bug for the wrong file path

* Fix no fetching correct artifact and test

* temporarily modify for test

* undo the change for test
2021-08-25 13:51:18 -07:00
Changming Sun
3837027506
Remove pyopenssl from installation (#8830) 2021-08-24 17:07:22 -07:00
liqun Fu
2beb873c6b
move training CI agent pools to 1ES hosted (#8775) 2021-08-18 18:36:19 -07:00
Chi Lo
51152e1aaa
Integrate TensorRT EP libs into existing GPU Nuget Package (Approach#1) (#8727)
* Merge CPU/GPU nuget pipeline

* Include TensorRT EP libraries into existing GPU nuget package pipeline

* modify to use correct YAML

* Modify for test

* modify for test

* Add depedance

* Add depedance (cont.)

* modify for test

* Add create TensorRT nuget package

* modify for test

* modify for test

* Merge CPU/GPU nuget pipeline

* Include TensorRT EP libraries into existing GPU nuget package pipeline

* modify to use correct YAML

* Modify for test

* modify for test

* Add depedance

* Add depedance (cont.)

* modify for test

* Add create TensorRT nuget package

* modify for test

* fix merge bug

* code refactor

* code refactor

* modify for test

* modify for test

* modify for test

* modify for test

* modify for test

* modify for test

* cleanup

* modify for test

* fix bug

* modify for test

* refactor

* fix bug and test

* Modify for test

* Modify for test

* Modify for test

* Modify for test

* Prepare for PR

* Prepare for PR

* code refacotr from review

* Remove naming 'Microsoft.ML.OnnxRuntime.TensorRT' to avoid confusion

* Add linux TensorRT libraries

* Remove redundant variable in YMAL

* revert file

* undo revert file

* Modify regular expression so that it can capture the correct file

* Remove newline at end of file

* small fix

* Revert to CUDA11.1 on Windows

* Add unit tests for nuget package on Linux

Co-authored-by: Changming Sun <chasun@microsoft.com>
2021-08-18 17:26:34 -07:00
Olivia Jain
60089f7093
Cuda11.4 (#8709)
* initial update from 11.1 to 11.4

* change 11.4.1 to 11.4.0

* adjusting to match nvidia/cuda image tags

* adjusting to match nvidia/cuda image tags centos7

* correction to 11.4.0

* correction to 11.4.0

* update to cuda 11.4

* change training back to 11.1

* change training back to 11.1

* point to correct nvcr.io/nvidia/cuda 11.4.1 image

* change centos8 to centos7

* correct cudnn path

* Update linux-gpu-ci-pipeline.yml for Azure Pipelines

* Update c-api-noopenmp-packaging-pipelines.yml

* need to resolve centos images but remove space and change to 11.4

* Update linux-gpu-ci-pipeline.yml

* add cudnn to docker image

* bump devtoolset to 10

* revert cuda 11.4 change to setup_env_trt

* orttraining back to 11.1

* use nvcr.io

* Fix previous change back to cuda 11.1

* update cudnn path

* use cudnn image (revert if failure)
2021-08-17 16:36:26 -07:00
Wei-Sheng Chin
47b3ecb53b
Packaging pipeline now builds with PythonOp (aka running autograd.Function) (#8652)
This PR disable UTs in training's package pipelines 
for building packages with PythonOp (torch.autograd.Function).
2021-08-17 10:55:13 -07:00
Changming Sun
ae6fdd3333
Bring code coverage dashboard back (#8394) 2021-08-16 20:54:39 -07:00
Changming Sun
f04a235c77
Update manylinux build scripts (#8724)
Update manylinux build scripts. Sync it with the latest upstream.
2021-08-13 12:04:00 -07:00
Changming Sun
5f74f198c1
Merge CPU/GPU nuget pipeline (#8683)
Merge CPU/GPU nuget pipeline. The old GPU nuget pipeline will be only for DML.

TODO: the result GPU package contains PDB files for some of the DLLs, but not all. It is due to the refactoring of CUDA EP to pluggable DLLs. At that time we forgot to copy the PDB files. However, I can't add them in now. Because currently the package is already 220MB large. If the missed PDB files were added, then it will be oversize. nuget.org doesn't accept >250MB packages.
2021-08-12 13:21:29 -07:00
Changming Sun
49a6ff75e6
Update py-packaging-stage.yml (#8569) 2021-08-02 09:17:15 -07:00
Changming Sun
0510688411
Update compliance tasks in python packaging pipeline and fix some compile warnings (#8471)
1. Update SDLNativeRules from v2 to v3. The new one allows us setting excluded paths.
2. Update TSAUpload from v1 to v2. And add a config file ".gdn/.gdntsa" for it.
3. Fix some parentheses warnings
4. Update cmake to the latest.
5. Remove "--x86" build option from pipeline yaml files. Now we can auto-detect cpu architecture from python. So we don't need to ask user to specify it.
2021-07-30 17:16:37 -07:00