Commit graph

868 commits

Author SHA1 Message Date
Changming Sun
6df4e293ff Remove unused code in tools/ci_build/github/azure-pipelines/nuget/templates/gpu.yml 2021-08-30 15:37:40 -07:00
Changming Sun
7cd46cb9c4 Fix a problem in Zip-Nuget-Java Packaging Pipeline 2021-08-30 14:51:36 -07:00
Edward Chen
b75c1081ca
[Objective-C] Enable static analysis, second try (#8875)
The previous attempt to enable static analysis (#8842) didn't actually run the static analysis checks.

- Run clang-tidy directly.
- Address static analysis warnings.
2021-08-30 10:43:45 -07:00
satyajandhyala
84f9271a8d
Enable registering external custom op schemas on Linux (#8889)
* Use manylinux instead of Ubuntu to run external custom ops build pipeline.
2021-08-30 10:13:47 -07:00
Changming Sun
1b5909dea8
Delete download_cmake.py (#8885) 2021-08-30 09:34:08 -07:00
satyajandhyala
31926176ac
Support external custom operator schemas on Ubuntu (#8807)
* Expose symbols in onnx and protobuf namespaces in python when building with --enable_external_custom_op_schemas

* Add external onnx and protobuf files to wheel

* Added an example to demonstrate external custom ops use-case

* Added a Linux build pipeline to test external custom ops
2021-08-28 11:05:21 -07:00
Guoyu Wang
6a1939252f
Fix Android java API failure (#8865)
* Fix Android Package break

* Without java fix -- pipeline should fail

* With java fix, should pass now

* address CR comments
2021-08-27 15:58:56 -07:00
Chi Lo
6a477acecf
Add tensorrt_provider_factory.h to artifact (#8869) 2021-08-27 09:09:54 -07:00
Yulong Wang
e8564d6597
[js/web] update emsdk to v2.0.26 (#8653)
* update emsdk to v2.0.26

* fix pooling build warning

* fix build break

* use pragma diagnostic semantic only when __GNUC__ is defined

* fix build break

* disable AttentionPastState_dynamic
2021-08-26 15:31:34 -07:00
Chi Lo
eb8f84e2a2
Fix issue of GPU tarball/zip/java package (#8850)
* modify for test

* modify for test

* modify for test

* modify for test

* modify for test

* modify for test

* prepare for PR

* Rename cuda directory to gpu directory in tarball

* Fix gpu java package

* fix bug

* fix small bug
2021-08-26 10:16:16 -07:00
Edward Chen
0cfc4ec09d
[Objective-C] Enable static analysis (#8842)
Add Objective-C API static analysis pipeline.
2021-08-26 09:13:52 -07:00
Changming Sun
ced2d8e597
Clean up TRT docker files (#8847) 2021-08-25 22:26:31 -07:00
Changming Sun
9cd7d836f7
Delete Dockerfile.ubuntu_for_android (#8848) 2021-08-25 22:25:14 -07:00
Guoyu Wang
613a600471
relax android ci timeout to 180 minutes (#8844) 2021-08-25 19:59:48 -07:00
Chi Lo
32ecbf4691
Create combined GPU tarball and zip file package (#8827)
* Add onnxruntime_providers_shared.dll into gpu nuget package

* Modify for test

* Temporarily remove for test

* Modify for test

* Modify for test

* Test packging Windows combined GPU

* Test packging Windows combined GPU

* Test packging Windows combined GPU

* Test packging Windows combined GPU

* modify for test

* modify for test

* fix bug

* Modify for test

* Modify for test

* Modify for test

* Modify for test

* Modify for test

* Modify for test

* Modify for test

* Modify for test

* Prepare for PR

* Prepare for PR

* Code refactor

* Rename proper Artifact name

* Rename intermediate Artifact names

* Revert Artifact Names

* Rename Artifact Names

* Modify Artifact name

* Modify Artifact name

* Modify Artifact name

* Update Java package

* Update Java package

* fix bug to change artifact name

* Fix bug for the wrong file path

* Fix no fetching correct artifact and test

* temporarily modify for test

* undo the change for test
2021-08-25 13:51:18 -07:00
Changming Sun
3837027506
Remove pyopenssl from installation (#8830) 2021-08-24 17:07:22 -07:00
Guoyu Wang
8992e31c85
Move iOS package from framework to xcframework (#8805)
* additional changes

* test package run

* minor fix

* minor fix

* minor fix

* Get around no arm64 simulator

* fix objc pod build failure

* downgrade_eigen

* update objc podspec template
2021-08-24 13:38:14 -07:00
Olivia Jain
4666a49106
Add Component Governance (#8794)
* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines
2021-08-20 17:41:18 -07:00
liqun Fu
2beb873c6b
move training CI agent pools to 1ES hosted (#8775) 2021-08-18 18:36:19 -07:00
pengwa
39059f2539
enable torch interop build (#8493)
* fix build - python.h not found

* disable --build_shared_lib for ortmodule tests

* fix

* fix the build flag

* disable --build_shared_lib for training path (not only for ortmodule)

* fix missing test model files

* disable test CApiTest.test_custom_op_library when ENABLE_TRAINING_TORCH_INTEROP is ON

* enable custom_op_library build

* fix build

* fix

* merge master and fix build failure

* build onnx_test_runner when onnxruntime_ENABLE_TRAINING_TORCH_INTEROP is ON

* resolve comments

* use --enable_training_torch_interop to replace "onnxruntime_ENABLE_TRAINING_TORCH_INTEROP=ON"
2021-08-19 09:16:32 +08:00
Chi Lo
51152e1aaa
Integrate TensorRT EP libs into existing GPU Nuget Package (Approach#1) (#8727)
* Merge CPU/GPU nuget pipeline

* Include TensorRT EP libraries into existing GPU nuget package pipeline

* modify to use correct YAML

* Modify for test

* modify for test

* Add depedance

* Add depedance (cont.)

* modify for test

* Add create TensorRT nuget package

* modify for test

* modify for test

* Merge CPU/GPU nuget pipeline

* Include TensorRT EP libraries into existing GPU nuget package pipeline

* modify to use correct YAML

* Modify for test

* modify for test

* Add depedance

* Add depedance (cont.)

* modify for test

* Add create TensorRT nuget package

* modify for test

* fix merge bug

* code refactor

* code refactor

* modify for test

* modify for test

* modify for test

* modify for test

* modify for test

* modify for test

* cleanup

* modify for test

* fix bug

* modify for test

* refactor

* fix bug and test

* Modify for test

* Modify for test

* Modify for test

* Modify for test

* Prepare for PR

* Prepare for PR

* code refacotr from review

* Remove naming 'Microsoft.ML.OnnxRuntime.TensorRT' to avoid confusion

* Add linux TensorRT libraries

* Remove redundant variable in YMAL

* revert file

* undo revert file

* Modify regular expression so that it can capture the correct file

* Remove newline at end of file

* small fix

* Revert to CUDA11.1 on Windows

* Add unit tests for nuget package on Linux

Co-authored-by: Changming Sun <chasun@microsoft.com>
2021-08-18 17:26:34 -07:00
Changming Sun
401de5911b
Remove CUDNN dir from setup_env_cuda.bat (#8762) 2021-08-18 10:32:57 -07:00
Rachel Guo
78759059f1
[CoreML EP]Make coreml ep build on non-macOS platform (#8677)
* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* clean

* remove unused defs

* correct typo

* remove onnxruntime_coreml_proto

* cr comments

* enablie nnapi/coreml in minimal build

* enable nnapi/coreml in one build

* refine dependencies

* fix nnapi build failure and remove onnxruntime_coreml_proto dependencies in unit tests cmake files

* small fix

* fix

* fix build

* revert

* fix build

Co-authored-by: rachguo <rachguo@rachguos-Mini.attlocal.net>
2021-08-18 09:35:32 -07:00
Olivia Jain
60089f7093
Cuda11.4 (#8709)
* initial update from 11.1 to 11.4

* change 11.4.1 to 11.4.0

* adjusting to match nvidia/cuda image tags

* adjusting to match nvidia/cuda image tags centos7

* correction to 11.4.0

* correction to 11.4.0

* update to cuda 11.4

* change training back to 11.1

* change training back to 11.1

* point to correct nvcr.io/nvidia/cuda 11.4.1 image

* change centos8 to centos7

* correct cudnn path

* Update linux-gpu-ci-pipeline.yml for Azure Pipelines

* Update c-api-noopenmp-packaging-pipelines.yml

* need to resolve centos images but remove space and change to 11.4

* Update linux-gpu-ci-pipeline.yml

* add cudnn to docker image

* bump devtoolset to 10

* revert cuda 11.4 change to setup_env_trt

* orttraining back to 11.1

* use nvcr.io

* Fix previous change back to cuda 11.1

* update cudnn path

* use cudnn image (revert if failure)
2021-08-17 16:36:26 -07:00
Wei-Sheng Chin
47b3ecb53b
Packaging pipeline now builds with PythonOp (aka running autograd.Function) (#8652)
This PR disable UTs in training's package pipelines 
for building packages with PythonOp (torch.autograd.Function).
2021-08-17 10:55:13 -07:00
liqun Fu
2b1f0816f8
to build cpu training packages for multiple multiple python versions (#8750)
Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
2021-08-17 10:49:44 -07:00
Changming Sun
ae6fdd3333
Bring code coverage dashboard back (#8394) 2021-08-16 20:54:39 -07:00
Dmitri Smirnov
8713d76dd1
Introduce C and C++ APIs for Sparse Tensors (#8621)
Add IsSparseTensor
  Add CreateSparseTensor
 Add utilities and test fully sparse instantiation
 Fully sparse blocksparse
 Add test and docs for fully sparse tensor instantiation
 Rework creation API
 Use API
 Non string API
 Retrofit of existing String API
 Add tests
 Add documentation
 Address build issues (Winml pending)
 Add inference test
 Bump binary size
 Add ifdef DISABLE CONTRIB
2021-08-16 16:33:47 -07:00
Changming Sun
8335d3dc0b
Fix Python Packaging Pipeline (Training Torch 1.9.0 Cuda 11.4) (#8738) 2021-08-16 14:46:43 -07:00
Olivia Jain
9cefd1303b
Integrate Anubis (#8603)
* copy over changes

* Update build_image.sh

* allow for configurable trt

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* reflect previous changes

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* model_list.json

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* checkout trt 7.1

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update post.py

* Update post.py

* Update post.py

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update model_list.json

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update post.py

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update post.py

* Update post.py

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update model_list.json

* Update post.py

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update post.py

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update post.py

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update start_job.ps1

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update run_mem_test_docker.sh

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Separate anubis files

* revert to old pipeline

* Update post.py

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* build off master Dockerfile

* Delete Dockerfile.custom-trt-perf

* Delete install_common_deps.sh

* uncomment

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml

* pass in trt container version

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update post.py

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update post.py

* remove sudo

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* add back build number

* allow python 3.8

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* python 3.8 fix trtexec

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* remove prev py38

* Update linux-gpu-tensorrt-ci-perf-pipeline.yml for Azure Pipelines

* add perf dependencies

* Update start_job.ps1
2021-08-16 13:20:28 -07:00
Changming Sun
f04a235c77
Update manylinux build scripts (#8724)
Update manylinux build scripts. Sync it with the latest upstream.
2021-08-13 12:04:00 -07:00
Edward Chen
76d21bbeb2
Update Android API level to 30. (#8717) 2021-08-12 18:01:13 -07:00
Changming Sun
5f74f198c1
Merge CPU/GPU nuget pipeline (#8683)
Merge CPU/GPU nuget pipeline. The old GPU nuget pipeline will be only for DML.

TODO: the result GPU package contains PDB files for some of the DLLs, but not all. It is due to the refactoring of CUDA EP to pluggable DLLs. At that time we forgot to copy the PDB files. However, I can't add them in now. Because currently the package is already 220MB large. If the missed PDB files were added, then it will be oversize. nuget.org doesn't accept >250MB packages.
2021-08-12 13:21:29 -07:00
liqun Fu
bec24ca4c1
create packaging pipeline to support cuda11.4 (#8663) 2021-08-11 17:44:57 -07:00
Guoyu Wang
a13daf550b
iOS Coacopods spec fix (#8678) 2021-08-11 10:11:44 -07:00
Edward Chen
20f006c580
Remove flake8 check from CMake build. (#8662) 2021-08-09 14:10:36 -07:00
Edward Chen
baf8c39a8d
Add Python checks pipeline (#7032)
This change adds a new pipeline for checking Python code. Currently this pipeline only runs flake8.
flake8 is also run as part of the CMake project builds, but we can switch over completely to the new pipeline later.
The .flake8 config file was also updated to make it easier to run standalone (flake8 --config ./.flake8) and some Python formatting issues were addressed in files that were not previously scanned.
2021-08-09 10:37:05 -07:00
Suffian Khan
6dd59a1117
revert onnx version (#8643) 2021-08-09 05:53:40 -07:00
liqun Fu
eab6c51413
to create a training cpu package for torch-ort documentation (#7845) 2021-08-05 16:43:37 -07:00
Changming Sun
0458821944
Delete linux-ort-srv-ci-pipeline.yml (#8628) 2021-08-05 15:06:07 -07:00
Ashwini Khade
96eb9810ba
Update onnx (#8458)
* updates for picking pnnx commit

* add tests filter to c# tests

* plus test fixes

* fix versioning for contrib ops

* fix tests

* test filter for optional ops

* more versioning related updates

* fix test

* fix layernorm spec

* more updates

* update docs

* add more test filters

* more filters

* update binary size threshold

* update docs

* plus more fixes

* updates per review

* update to release commit

* add filters for optional type tests

* plus updates
2021-08-05 09:21:44 -07:00
Edward Chen
1041fa34f4
Specify timeout for iOS packaging pipeline (#8616) 2021-08-04 11:17:50 -07:00
Changming Sun
6c69baf78e
Disable Training Windows GPU Debug build because it is failing (#8608) 2021-08-04 09:24:27 -07:00
Edward Chen
717627775a
Increase build timeout (#8583) 2021-08-02 14:50:01 -07:00
stevenlix
ee99fb400c
Upgrade TensorRT to v8.0.1 (#8512)
* update onnx-tensorrt parser to master

* disable unsupported tests

* add cuda sm 75 for T4

* update tensorrt pipeline

* update trt pipelines

* update trt pipelines

* Update linux-gpu-tensorrt-ci-pipeline.yml

* update trt cid pipeline

* Update linux-gpu-tensorrt-ci-pipeline.yml

* Update Tensorrt Windows build pool and TensorRT/CUDA/CuDNN version

* update to cuda11.4 in trt ci pipeline

* update base image to cuda11.4

* update packaging pipeline to cuda11.4

* clean up

* remove cuda11.1 and cuda11.3 docker file

* disable unsupported tensorrt tests at runtime

* Update linux-multi-gpu-tensorrt-ci-pipeline.yml
2021-08-02 11:20:31 -07:00
Changming Sun
49a6ff75e6
Update py-packaging-stage.yml (#8569) 2021-08-02 09:17:15 -07:00
Changming Sun
0510688411
Update compliance tasks in python packaging pipeline and fix some compile warnings (#8471)
1. Update SDLNativeRules from v2 to v3. The new one allows us setting excluded paths.
2. Update TSAUpload from v1 to v2. And add a config file ".gdn/.gdntsa" for it.
3. Fix some parentheses warnings
4. Update cmake to the latest.
5. Remove "--x86" build option from pipeline yaml files. Now we can auto-detect cpu architecture from python. So we don't need to ask user to specify it.
2021-07-30 17:16:37 -07:00
Scott McKay
c6f95841dc
Add HardSigmoid to mobile packages. Used by PyTorch MobileNet v3 (#8552) 2021-07-30 12:08:11 +10:00
Guoyu Wang
464fd28ee9
Update iOS packaging script to default build static framework, disable bitcode (#8533)
* default package build to static, disable bitcode

* fix pipeline failure

* Address CR comments
2021-07-29 17:28:02 -07:00
Ye Wang
ad093b94b9
Restore transformers tests and disable some tests (#8530)
* restore transformers tests and disable some tests

* test

* update

* pass pep8 check

* update
2021-07-29 14:09:36 -07:00