Commit graph

574 commits

Author SHA1 Message Date
baijumeswani
93bf7c4d52
Documentation for distributed CI tests pipeline (#6140) 2021-01-04 10:09:39 -08:00
Changming Sun
1685167e46
Update manylinux docker image to the latest (#6242) 2020-12-31 19:57:04 -08:00
Xavier Dupré
cd14c1af29
Support double for operator ArgMin (#6222)
* Support double for operator ArgMin
* add test specifically for double
* add new test on pai-excluded-tests.txt
2020-12-31 11:25:46 +01:00
Xavier Dupré
84addcd2cf
Support double for operator ReduceMean, ReduceLogSumExp (#6217)
* Support double for operators ReduceMean, ReduceLogSumExp
2020-12-31 11:24:54 +01:00
Changming Sun
3911105f09 Remove python 3.5 2020-12-30 20:16:45 -08:00
Changming Sun
1b23b28706
Remove MKLML/openblas/jemalloc build config (#6212) 2020-12-30 17:18:19 -08:00
sfatimar
7347996942
Openvino ep 2021.2 (#6196)
* Enabling fasterrcnn variant and vehicle detector

* changes for 2021_2 branch

* yolov3_pytorch commit

* fixed braces in basic_backend.cc

* ci information added

* faster rcnn variant and vehicle detector changes were made in 2021.1 and not in 2021.2

* some changes to support unit tests

* disable some tests which are failing

* fix myriad tests for vehicle detector

* Did some cleanup
*cleaned up comments
*Disabled Add_Broadcast_0x1 and Add_Broadcast_1x0
tests on MYRIAD_FP16 backend due to a bug
*cleaned up capability_2021_2.cc file
*Removed extra conditions which were added
for some validation in backend_utils

Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>

* yolov3 pytorch workaround to ensure that the output names are matched

* gemmoptest fixed on myriad

* Fixed MYRIADX CPP Test Failures

*Expand,GatherND,Range,Round op's
are only supported in model

*where op with float input data
types are not supported and fixed

*Scatter and ScatterElements op's with
negative axis are fixed

*Reshape op with 0 dim value are not
supported and fixed

*Disabled InstanceNorm_2 test on MYRIADX

Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>

* make changes to yolov3 pytorch

* Fixed python unit tests
*Fixed failing python tests on vpu,
GPU and CPU

Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>

* Fixes POW op failures on GPU_FP16

Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>

* Clean up capability_2021_2.cc

Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>

* Updated docx for MultiThreading option
*Added extra info on setting the num_of_threads
option using the API and it's actual usage

Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>

* fixed slice and removed extra prints

* Disabled failing python tests

Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>

* Minor changes added in capabilty_2021_2

Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>

* made changes to slice to avoid failures

* Disabling FP16 support for GPU_FP32
->Inferencing an FP16 model on GPU_FP32
leads to accuracy mismatches. so, we would
rather use GPU_FP16 to infer an FP16 model
on GPU Device

Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>

* Updated docx for Inferencing a FP16 Model

Signed-off-by: MaajidKhan <n.maajidkhan@gmail.com>

* fix for mask rcnn

* Script for installing openvino from source

* Updated with openvino 2021.2 online installation

* code comment fixes
fixed accuracy mismatch for div

* Update OpenvinoEP-ExecutionProvider.md

updated for 2021.2 branch

* Update README.md

updated dockerfile documentation

* Update BUILD.md

build.md update documentation

* permissiong change of install_openvino.sh

* made changes to align with microsoft onnxruntime changes

* Updated with ov 2021.2.200

Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
Co-authored-by: sfatimar <sahar.fatima@intel/com>
Co-authored-by: MaajidKhan <n.maajidkhan@gmail.com>
Co-authored-by: mohdansx <mohdx.ansari@intel.com>
2020-12-23 08:47:22 -08:00
satyajandhyala
201d0dbb1a
Android coverage dashboard (#6163)
* Write the report to a file.

* Post code coverage to the Dashboard database.
2020-12-21 10:34:01 -08:00
Edward Chen
cd3a5acca0
Update get_docker_image.py to enable use without image cache container registry. (#6177)
Update get_docker_image.py to enable use without image cache container registry.
2020-12-18 19:01:02 -08:00
Changming Sun
344a2a8ee5
Revert "work around of the build break in mac (#6069)" (#6150)
This reverts commit 3cae28699b.
2020-12-16 14:41:18 -08:00
Sheil Kumar
a6a23db130
Enable C# .NET5 for WinML (#6120)
* build for .net5

* only reference cswinrt for .net5

* remove netstandard2.0 references

* upgrade language version

* net5

* remove extra comment closure

* add targetframework

* set target framework

* remove net*

* pep8 errors

* make test project build with .net windows SDK projection

* disable c# builds for non-x64 builds

* fix pep8 errors

* disable for store build

* fix tests

* remove cswinrt and sdk references from package

* bump cswinrt down to 1.0.1

* fix bin path

Co-authored-by: Sheil Kumar <sheilk@microsoft.com>
2020-12-14 15:05:15 -08:00
liqunfu
cde723a136
Liqun/move nightly pl to linux multi gpu v100 (#6024)
* move e2e nightly pipeline to azure devop
Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
2020-12-14 12:43:41 -08:00
Vincent Wang
7ddeafdfcc
Add ReduceL2Grad and ClipGrad (#5970)
* ReduceL2Grad and ClipGrad.

* fix win build and amd ci pipeline

* resolve comments.

Co-authored-by: Vincent Wang <weicwang@AiFramework2080ti2.corp.microsoft.com>
2020-12-10 11:03:26 +08:00
Edward Chen
e357486707
Fix build definition template typo, add logging (#6065)
Fix a typo in tools/ci_build/github/azure-pipelines/templates/get-docker-image-steps.yml.
Add logging to tools/ci_build/get_docker_image.py for easier debugging.
2020-12-08 15:16:50 -08:00
satyajandhyala
f68a256140
Android code coverage (#6061)
* Added Onnxruntime_GCOV_COVERAGE flag for Android.

* Set CMAKE_SYSTEM_NAME explicityly for Android.

* Added GCOV_PREFIX option to collect code coverage data.
Added a new python script to generate code coverage info.
Modified build pipeline to geneate Android code coverage info

* Added build command line option --android_coverage

* Added a comment describing the GCOV environment variables

* Fixed PEP8 issues.

* Added --android_coverage option to the build command.

* Increased Android emulator memory from 3K to 8K.

* Increased Android partition-size from 2GB to 4GB to overcome no-space-left-on-device error

* Removed source_dir from command line args.

* Use cwd absolute path to run tests.

* Added commands to output the contents of /data/local/tmp on the emulator.

* Added run_adb_shell function.

* Format changes.

* Removed keywd argument cwd.

* Removed Android in the --build_dir path.

* Removed commands added for debugging.

* Removed exxtra new-lines.

* Fix MacOs build pipeline failures by uninstalling openssl before running build script.

* Revert "Fix MacOs build pipeline failures by uninstalling openssl before running build script."

This reverts commit 90d0568fe533e9456c20d061a2d435c8fea48266.

* Change dir to the build directory where the tar file is copied.

* Changed the option from --android_coverage to --code_coverage

* Moved steps to generate Android code coverage to run_nnap_code_coverage.sh

* Require --android option if --code_coverage is specified.

* No code coverage needed for onnx_test_runner.

* Expect that the emulator is running when the script is executed.

* Fixed the title in the buildpipeline step.

* Fixed the formatting issue.

* Added a command line argument, ORT_ROOT, to run_nnapi_code_coverage.sh script

Co-authored-by: Satya Jandhyala <satyajandhyala@Satyas-Mac-mini.local>
2020-12-08 10:55:02 -08:00
Suffian Khan
e35211c0ff
Fix AMD GPU pipeline by adjusting reference /opt/rocm-3.9.0 => /opt/rocm (#6063)
* use /opt/rocm instead

* fix indent
2020-12-08 08:53:20 -08:00
Yufeng Li
3cae28699b
work around of the build break in mac (#6069)
* Fix the build break in macos release

* revert android change
2020-12-07 20:39:36 -08:00
Edward Chen
b348538c8a
Update build docker image cache cleanup (#6048)
The current image cache cleanup is not removing many images. Upon examining the cache container registry logs, it appears there are some infrequent pulls of old images which may be made by something other than CI builds (perhaps some automated scan of the registry).
This change adds a minimum access count for images in the cache so that infrequently but periodically accessed images can be removed. The idea is that images used by CI builds that are worth caching will have a higher volume of accesses.
2020-12-07 13:07:19 -08:00
Changming Sun
925879a8b0
Remove python 3.8 Windows GPU build from python packaging pipeline (#6054)
Revert the last a few changes to get the pipeline back to a normal state.
2020-12-07 10:23:07 -08:00
Edward Chen
d8139814fd
Clean up builds (#6015)
Update training Python packaging build to use get_docker_image.py.
Remove BUILD_EXTR_PAR docker build argument.
Update get_docker_image.py to check again for the image in the cache after building and before pushing to reduce the chance of a redundant push.
2020-12-04 15:13:17 -08:00
Jesse Benson
98ea7372d3 Re-enable Lamb unit tests for AMD 2020-12-03 13:06:34 -08:00
Edward Chen
6572a4d306
Disable Python 3.9 for training Python packaging build. (#6012)
Disable Python 3.9 for training Python packaging build. Python 3.9 is not supported by the PyTorch dependency.
2020-12-03 11:42:28 -08:00
Edward Chen
6d642a3dba
Replace direct pulls from image cache container registry with get_docker_image.py, build definition clean up. (#5906) 2020-12-01 19:10:23 -08:00
Changming Sun
2d9dcc4576
Add python 3.9 support (#5874)
1. Add python 3.9 support(except Linux ARM)
2. Add Windows GPU python 3.8 to our packaging pipeline.
2020-11-30 12:02:48 -08:00
Changming Sun
5fdd9f0fd2
Fix Python Linux GPU package name (#5943)
Fix Python Linux GPU package name. I accidentally added "noopenmp" to it.
2020-11-25 17:46:11 -08:00
Edward Chen
7546d251e0
Expose parameters in clean build Docker image cache build. (#5941)
Expose some parameters in the clean build Docker image cache build. In particular, whether to do a dry-run and the lifetime of unused cache images.
2020-11-25 14:15:54 -08:00
Ashwini Khade
705d093167
Update onnx (#5720)
* update onnx

* update docker image for testing
2020-11-24 11:20:15 -08:00
Suffian Khan
9b8189dd0a
Rework AMD CI pipeline to use pool AMD-GPU and disable more tests in order to enable it. (#5885)
Move AMD test pipeline to use self-hosted pool AMD-GPU. For time being, remove failing/flaky unit tests for AMD pipeline.
2020-11-24 09:38:14 -08:00
Guoyu Wang
846c5fb917
Report arm64 minimal baseline binary size only for continuous integration (#5913)
* Report binary size only for continuous integration
2020-11-24 20:24:08 +10:00
Guoyu Wang
4137c18d9b
Add ORT minimal with NNAPI EP to Android CI (#5890)
Description: Add ORT minimal with NNAPI EP to Android CI

Motivation and Context

The added build/test to Android CI will only run UT, additional onnx_test_runner with customer .ort models will be added later
2020-11-23 18:21:34 -08:00
Edward Chen
5e8fcda24a
Build docker image cache fixes. (#5902)
Fix Python 3.5 compatibility issue in tools/ci_build/get_docker_image.py.
Fix line endings in tools/ci_build/github/azure-pipelines/clean-build-docker-image-cache-pipeline.yml.
2020-11-23 14:43:12 -08:00
baijumeswani
208f4c1d3c
Azure ci pipeline for distributed environment tests (#5881) 2020-11-23 14:01:00 -08:00
satyajandhyala
353e071b7e
Fuzz testing misc (#5862)
* Run only required steps relevant to fuzz testing.

* Exit status non-zero for any uncaught exception other than ort_exception in the driver code
Co-authored-by: Satya Jandhyala <sajandhy@microsoft.com>
2020-11-23 13:43:44 -08:00
Guoyu Wang
26e6ced172
Temporary fix for Android CI failure (#5889)
* Unblock the Android CI

* Add python to android ci's command
2020-11-21 17:58:32 +10:00
Dmitri Smirnov
ceedf5630b
Document all C# API pubic interfaces (#5853)
Address documentation shortcomings.
 Document all required public interfaces.
 Add pipeline configuration.
Make Doxygen lookup a env vars for paths.
2020-11-20 14:03:55 -08:00
Edward Chen
bef06dac93
Automatically clean up build docker image cache. (#5843)
Follow up to #5811 to automate cleanup of the build docker image cache.
Added a script and build definition to clean up docker images that haven't been accessed recently.
2020-11-20 11:56:26 -08:00
S. Manohar Karlapalem
ff58f621fa
Remove nGraph Execution Provider (#5858)
* Remove nGraph Execution Provider

Pursuant to nGraph deprecation notice: https://github.com/microsoft/onnxruntime/blob/master/docs/execution_providers/nGraph-ExecutionProvider.md#deprecation-notice

**Deprecation Notice**

| | |
| --- | --- |
| Deprecation Begins	| June 1, 2020 |
| Removal Date |	December 1, 2020 |

Starting with the OpenVINO™ toolkit 2020.2 release, all of the features
previously available through nGraph have been merged into the OpenVINO™
toolkit. As a result, all the features previously available through
ONNX RT Execution Provider for nGraph have been merged with ONNX RT
Execution Provider for OpenVINO™ toolkit.

Therefore, ONNX RT Execution Provider for **nGraph** will be deprecated
starting June 1, 2020 and will be completely removed on December 1,
2020. Users are recommended to migrate to the ONNX RT Execution Provider
for OpenVINO™ toolkit as the unified solution for all AI inferencing on
Intel® hardware.

* Remove nGraph Licence info from ThirdPartyNotices.txt

* Use simple Test.Run() for tests without EP exclusions

To be consistent with rest of test code.

* Remove nGraph EP functions from Java code
2020-11-19 16:47:55 -08:00
Hariharan Seshadri
62508ef0e4
Revert "Remove MKLML build config (#5559)" (#5855) 2020-11-19 10:53:08 -08:00
Changming Sun
26db396b4b
Reduce the number of CI build variants (#5856) 2020-11-18 20:41:30 -08:00
satyajandhyala
b495ae8103
ORT fuzz testing (#5771)
* Added fuzz testing using ORT model.

* The onnxruntime_security_fuzz driver code should accept either ONNX or ORT (based on the file extension) input file if /f flag is provided.

*  Added ValidateOrtFormatModelDoesNotRunOptimizersInFullBuild test.

* Added win-ci-fuzz-testing.yml to run build pipeline.

* Prevent out-of-range access in the graph.cpp.
2020-11-18 16:07:36 -08:00
Changming Sun
85f945a875
Regenerate CI build docker images (#5850) 2020-11-18 14:36:59 -08:00
Edward Chen
71e7c2b423
Cache build docker images in container registry. (#5811)
This PR adds infrastructure to automatically cache docker images used in CI builds in a container registry.

Currently, build images are pulled from a container registry for some builds and built every time for others. The container registry requires maintenance to keep the images up to date and building images every time wastes build agent resources.

With this change, a given build image can be looked up in a cache container registry and if present, pulled, and otherwise, built and pushed. The uniqueness of a build image is determined by a hash digest of the dockerfile, docker build context directory, and certain "docker build" options. This digest is part of the image tag in the cache container repository.

The cache container registry will need to be cleaned up periodically. This is not automated yet.
2020-11-17 17:02:24 -08:00
Justin Stoecker
bd236ecc26
Switch to unified DirectML 1.4.0 redistributable (#5794)
Transitions from the ORT-only DML NuGet (hosted on the onnxruntime_public feed) to the new unified DirectML NuGet (Microsoft.AI.DirectML) on nuget.org. In addition, the Microsoft.AI.MachineLearning (WinML) and Microsoft.ML.OnnxRuntime.DirectML packages now take a dependency on the Microsoft.AI.DirectML package. This means we can remove the extra copy of DML binaries in these packages since they will be installed by the DML package.
2020-11-17 13:42:23 -08:00
Scott McKay
c84bc25e28
Add validation of op registrations (#5817)
* Add validation of operator registrations to the reduction script
  - the script has all the logic to process the registrations, and there's a CI that uses it

Fix some operator registrations

* Fix CUDA PRelu registration

* Refactor to split out kernel registration file parsing and use in the exclude ops script and an op registration validation script.
Run op validation in minimal build CI

* Fix PEP8 error and some comments
2020-11-17 10:44:09 -08:00
Sherlock
241b2226a7
Update orttraining-linux-gpu-ci-pipeline.yml for Azure Pipelines (#5826) 2020-11-17 09:27:59 -08:00
Guoyu Wang
1a66dfc0f9
Enable Squeeze Opset 13 for NNAPI (#5717)
* Add copy sparse model in minimal CI

* Add squeeze 13 support

* fix small typo

* Add ut for squeeze in NNAPI

* Fix some issue in the UT and code

* Modify based on the master change

* Fix build break
2020-11-17 00:26:06 -08:00
edgchen1
4d517c68a3
Fix reference to old download_e2e_test_data.py script. It was renamed to download_azure_blob.py. (#5790) 2020-11-12 15:48:06 -08:00
liqunfu
1416d12f0b
Liqun/merge e2e pipelines (#5702)
* Create an Azure Pipeline to merge cpp and python e2e pipelines into one. Still keep cpp 2e2 pipeline until this new pipeline is stable.

Co-authored-by: liqun <liqun@OrtTrainingDev4.af05slrtruoetgaxwwjv5nsq5e.px.internal.cloudapp.net>
2020-11-11 09:42:08 -08:00
Changming Sun
79350a642a
Update install_deps.sh: remove the unnecessary data generating step (#5758)
We install onnx python package from this script, so python tests can run the tests for the latest commit which we are importing.
2020-11-10 22:19:03 -08:00
Changming Sun
4094a09a56
Merge pull request #5731 from microsoft/snnn/rtti
Disable RTTI in Windows GPU CI pipeline
2020-11-10 09:02:59 -08:00