Commit graph

98 commits

Author SHA1 Message Date
Changming Sun
deea945f80
Remove openmp and scipy from build pipelines (#4305)
1. Remove openmp because the default thread pool is already good enough.
2. Remove scipy from build pipelines because it stops support python 3.5.
2020-06-23 20:18:16 -07:00
edgchen1
737c22a911
Refactor Python packaging builds (#4283)
Reuse the same template file for all Python packaging builds.
2020-06-22 17:13:22 -07:00
Changming Sun
0349479b19
Fix component governance and codesign validation errors (#4277)
Adjust the job steps so that these security tasks run before the build directory clean up.
2020-06-18 15:54:18 -07:00
Changming Sun
6f4320fb85
Fix the python package name issue (#4207)
Fix the package package name issue. In my last change(#4197) about enabling code sign. I forgot to pass the additional flags to setup.py,
2020-06-12 08:32:59 -07:00
Dmitri Smirnov
af0750ba1b
Java GPu artifact naming (#4179)
Modify gradle build so artifactID has _gpu for GPU builds.
  Pass USE_CUDA flag on CUDA build
  Adjust publishing pipelines to extract POM from a correct path.

Co-Authored-By: @Craigacp
2020-06-10 11:15:48 -07:00
Sheil Kumar
4377ff4a1a
Enable .NET Core 2.0 and .NET Framework 4.6.1 in Microsoft.AI.MachineLearning NuGet package (#4125)
* add project to download cswinrt and build winrt c# interop dll

* Add to nuget package

* reverse if check

* run generation before core compile

* add generated files to compile

* update .net package to binplace native libs

* add props to .netstandard2.0 folder

* auto binplace ml native binaries

* force 'Any CPU' platform build

* Fix anycpu and platform targets

* fix flake errors

* fix variable order

* fix flake pep8 errors, semicolon

Co-authored-by: Sheil Kumar <sheilk@microsoft.com>
2020-06-09 09:08:19 -07:00
Yulong Wang
842be1535d
[Node.js binding] add linux and mac package (#4157)
* try mac pipeline

* fix path separator

* copy prebuilds folder

* split esrp yaml for win/mac

* disable mac signing temporarily

* add linux

* fix indent

* add nodetool in linux

* add nodetool in win-ci-2019

* replace linux build by custom docker scripts

* use manylinux as node 12.16 not working on centos6

* try ubuntu

* loosen timeout for test case - multiple runs calls
2020-06-08 14:12:05 -07:00
Yulong Wang
2aab20b4ea
[Node.js binding] upgrade node-addon-api to 3.0 (#4148) 2020-06-05 21:24:34 -07:00
Yulong Wang
2e58097f8f
fix build: pipeline Node.js version to 12.16.3 (#4145) 2020-06-05 17:56:03 -07:00
Yulong Wang
647a886587
[Nodejs binding] create a new pipeline to generate signed binaries (#4104)
* add yml files

* update pipeline

* fix yaml syntax

* yaml pop BuildCSharp

* udpate yaml

* do not stage codesign summary
2020-06-02 01:28:05 -07:00
Dmitri Smirnov
afca0d15ee
Create Java publishing pipeline (#3944)
Create CPU and GPu Java publishing pipelines. Final jars are tested on all platforms. However, signing and publishing to maven are manual steps.
2020-06-01 18:18:57 -07:00
edgchen1
a715d55bcc
Training Python package fixes (#4063)
- Add support for ENABLE_LANGUAGE_INTEROP_OPS in training build which is enabled for nightly builds
- Fix passing of environment variables to `sudo docker run` in build definitions
- Fix setup.py package naming logic
2020-06-01 09:30:56 -07:00
Yulong Wang
b3ec8035ee
[Node.js binding] add build flag for node.js binding (#3948) 2020-05-27 13:30:22 -07:00
Changming Sun
2fa2019daf
Run docker commands with sudo (#3979) 2020-05-18 17:35:09 -07:00
edgchen1
e259a13f8e
Initial training Python packaging pipeline (#3767)
Add a pipeline to produce training-enabled ORT wheels.
2020-05-18 09:41:00 -07:00
edgchen1
e55f24364a
Disable LTO on Windows training CPU build (#3960)
Disable LTO on Windows training CPU build. Add a parameter to the win-ci-2019.yml build template for enabling LTO with a default value of true.
2020-05-18 09:24:10 -07:00
Ori Levari
7b858d60b0
Various changes for automated downlevel test pipeline (#3901)
Co-authored-by: Ori Levari <orlevari@microsoft.com>
2020-05-12 17:22:47 -07:00
stevenlix
4ea10c9202
bump up ORT version and extend time limit for windows cpu packaging pipelines (#3852) 2020-05-07 14:22:20 -07:00
Sheil Kumar
37b60251ca
test packaging (#3756)
Co-authored-by: Sheil Kumar <sheilk@microsoft.com>
Co-authored-by: Changming Sun <chasun@microsoft.com>
2020-05-02 12:23:33 -07:00
Scott McKay
9f72752397
Fix 'Install ONNX' CI failure (#3761)
* Disable flaky test temporarily

* turn off pip upgrade warning

Co-authored-by: Aishwarya Bhandare <aibhanda@microsoft.com>
Co-authored-by: Zeeshan Siddiqui <mzs@microsoft.com>
2020-04-30 18:18:58 +10:00
Ori Levari
78fde2c4cb
add downlevel test artifact to windowsai-nuget build (#3711) 2020-04-28 10:05:32 -07:00
edgchen1
e22d97ba56
Merge pull request #3643 from microsoft/ort_training_for_merge_to_master
Introduce ORT training implementation
2020-04-25 07:15:22 -07:00
Sheil Kumar
a475f2824d
Create the Nuget WindowsAI Pipeline (#3684)
* add windowsai.yml for new Microsoft.AI.MachineLearning nuget

* temporarily add windowsai.yml to gpu.yml

* pass in build arch

* remove install onnx task

* no dml for arm or arm64

* refactor nuget pipeline defs

* update package creation

* pass in build and sources path

* missing hyphens

* copy license file

* fix parameter variable

* disable arm builds for now

* remove commented script block

* download pipeline atifcat name update

* set working dir

* Add bundling nuget script

* path combine

* null path

* combine needs parentheses

* binplace microsoft.* dlls in new nuget package

* update artifact name

* move merged nuget to artifacts directory

* move to merged subfolder in artifacts staging dir

* forward slash to back

* enable arm

* vcvarsall needs x64 vars setup

* Run Tests

* fix tests

* move global variables

* update yml to not have global variable in template

* removed parameters

* fixes

* Add build arch as an env variable

* ne not neq

* %Var% for batch script

* dont pass argument for x64

* disable arm tests

* skip csharp/cxx tests for microsoft nuget package

* remove test-win as it tests only c# cxx and capi

* test build for store apps

* dont build for store

* tools/nuget/generate_nuspec_for_native_nuget.py

* remove args.

* add new props and targets for microsoft.ai

* make windowsai props/targets static

* add dependency

* dont ship dot net props

* Remove c# fom windowsai nuget

* copy license file

* native packages must have win10 as the platform, not win

* cuda header in wrong if branch

* no dml for arm builds

* only build dml for x64/ x86

* User/sheilk/props update (#3616)

* prelim store work

* props

* Fix desktop nuget props/targets

* clean up targets and make store apps work

Co-authored-by: Sheil Kumar <sheilk@microsoft.com>

* update windowsai.yml with latest

* remove extra dloadhelpers

* Add abi headers to abi dir, and reference native includes

* update windowsai.yml

* minor update

* remove parameters

* add doesrp param

* hard code esrp to true

* add directml for x86/x64

* revert gpu yml changes

* add store builds

* add store builds

* add checks again in old way

* dup job names for store and desktop builds

* move all of the runtime binaries to win10 folder

* only set safeseh on x86

* disable the store builds for now... missing msvcprt.lib

* copy paste deletion...

* switch back to win- (#3646)

Co-authored-by: Sheil Kumar <sheilk@microsoft.com>

* use stahlworks

* & not supported in ado

* add cuda to cpu nuget(???) and EnableDelayedExpansion to enable x86 dml package

* revert nocontribops

* add underscore...

* extra win/win10 change

* merged nuget... still not being bundled...

* files in merged directory

* missing parens causing dml to be included in cpu package

* more diagnostic info

* switch dir to get-childitem

* wait for compression to complete

* add winml_adapter to mkml and gpu packages

* enable_wcos

* add mklml binaries

* props and targets missing from mklml

Co-authored-by: Sheil Kumar <sheilk@microsoft.com>
2020-04-24 20:20:04 -07:00
Edward Chen
daa14b64e3 Merge remote-tracking branch 'origin/master' into edgchen1/merge_from_master 2020-04-21 03:31:32 +00:00
Changming Sun
d68245853e
Disable downloading test data on Linux (#3581) 2020-04-18 15:54:58 -07:00
Changming Sun
1a222b3f6e
Disable downloading test data on Windows (#3551)
* Disable downloading test data on Windows
2020-04-16 22:15:20 -07:00
harshitha
80e0c64e2e merged with master 2020-04-16 17:13:36 +00:00
Changming Sun
7c89f38a34
Fix static analysis warnings found by VC++ (#3530)
1. Fix static analysis warnings found by VC++
2. Add a new pipeline for static analysis
3. Merge all the windows CI build into one single yaml file.(Easier to queue them all).
4. Make DNNL build faster by disabling building the tests and examples.
5. Enable custom op unitest.
2020-04-16 01:46:47 -07:00
David Brownell
006c5be1b1
Optionally produce a python wheel that includes featurizers (#3491) 2020-04-14 09:00:13 -07:00
M. Zeeshan Siddiqui
5d99f179b9
Merge pull request #3486 from microsoft/sedymche/merge_master_ort_training
Merge from master into ort_training
2020-04-13 10:55:36 -07:00
Sergii Dymchenko
bf3df41424 Put back SubmoduleCheckoutMode parameter into mac-ci.yml. 2020-04-12 21:49:38 -07:00
edgchen1
cffdff6702
Publish unit test results from Linux and Mac builds (#3480)
* Added publish test results step to Linux and Mac builds.

* Fix test result file pattern.
2020-04-10 14:51:56 -07:00
Sergii Dymchenko
6ba7c99e50 Merge branch 'master' into ort_training 2020-04-09 12:42:04 -07:00
Changming Sun
a5fea26cb4 Disable model tests for Mac OS X builds 2020-04-02 15:14:32 -07:00
Xueyun Zhu
ccc3535e72 resolve conflict 2020-03-20 20:20:35 +00:00
Tiago Koji Castro Shibata
3bdb0b620a
Fix WCOS/Win32 linking bugs (#3126)
* Fix WCOS/Win32 linking bugs

* Remove unused NODEFAULTLIB flags

* Avoid plain target_link_libraries signature

* Avoid plain target_link_libraries signature

* Fix library list escaping

* Use library list instead of string

* Remove duplicate link to windowsapp.lib

* Remove Win32 build workarounds

* Specify CMake policies before initializing language

* Expose Win32 header definitions during build

* Force set API family

* Enable Win32 APIs in featurizer

* Use MT dynamic CRT

* Expose Win32 specific functions

* Disable app container globally

* Disable default wide functions in featurizers

* Add featurizers to test include path

* Workaround https://gitlab.kitware.com/cmake/cmake/issues/19428

* Revert pipeline debugging hacks

* Skip /FI in CUDA sources

* Default to Win32 builds

* Enable WCOS when using WinML

* Use generator expression to apply CMAKE_MSVC_RUNTIME_LIBRARY to C++ only
2020-03-19 08:52:40 -07:00
Edward Chen
80dd62a240 Enable CI for training. 2020-03-11 14:41:32 -07:00
Edward Chen
e542cfd0e0 Introduce training changes. 2020-03-11 14:39:03 -07:00
Tiago Koji Castro Shibata
a59243090a
Publish release symbols (#3152)
* Publish release symbols

* Publish symbols if IsReleaseBuild
2020-03-05 22:32:18 -08:00
smk2007
6cdd2b4934
Enable DML Nuget Package for x64 or x86 architectures (#3120)
* add dml gpu pipelines

* add x86 to the gpu dml dev build pipeline

* Enable DML x86 builds

* Fix uint64_t -> size_t warning

* fix warnings

* enable dml on x86 ci builds

* operatorHelper 773 error uint32_t vs uint64_t

* operatorHelper 773 error uint32_t vs uint64_t

* make x86 pipeline use the gpu pool

* more warnings

* fix x86 directml path

* make dml nuget package

* disable tf_pnasnet_large

* disable zfnet512

* make validation use wildcards

* disable x86 dml gpu tests

* add args.

* update gpu.yml

* change nupkg wildcard

* add debug statements

* package x86 dml nupkg

* dont drop managed nuget again from dml pipeline build

* Add DML EULA

* directml license should be renamed to not clobber the existing license

* casing on dml package....

* {} to ()

* fix license name

* disable dml from x86 ci

* typo and cr feedback

* remove featurizers

* ship the dml pdb as well
2020-03-02 20:18:46 -08:00
Hariharan Seshadri
d7f2cdcc7e
Fix target platform of managed OnnxRuntime dll and enable x86 .NET testing (#3056)
* WIP: Re-enable x86 .NET testing in Release pipelines

Enabling x86 testing will make sure that ORT packages doesn’t break x86 projects of customers

* Remove setting some env variables

* Comment out a test failing on x86 builds

* More changes

* Minor fix

* More changes

* More changes

* s

* s

* s

* Revert minor change

* More changes

* More changes

* More changes 2

* explicitly set platform target

* Delete bin and obj folders

* Clean output dirs

* Add back TargetFramwork

* Disable x86 .net framework tests

* Skip x86 tests in MKLML pipeline
2020-02-24 23:02:59 -08:00
Changming Sun
fb871978b5
Adjust build flags for the release pipelines (#3066)
1. Add LTCG back. It was set to default OFF in my previous PR to speed up Windows build. It is only needed in release pipelines.
2. Remove --use_featurizers from all the packaging pipelines
3. Make sure all the packages have openmp
2020-02-21 16:45:42 -08:00
Changming Sun
179603775f
Use CUDA 10.1 for Linux build (#3057)
Use CUDA 10.1 for Linux build
(Windows change is already in)

Please note, cublas 10.2.1.243 is for CUDA SDK 10.1.243, not CUDA 10.2.x. CUDA 10.2.89 need cublas 10.2.2.89. They match on the last part of the digits.

libcublas10-10.1.0.105 won't work!!!

The cuda docker image by viswamy is already using 10.1, no need to change.
2020-02-21 11:55:32 -08:00
Tiago Koji Castro Shibata
fb2182f3fc
Release ARM/ARM64 Nuget packages (#2987)
* Enable ARM64 release builds

* Add ARM release

* Skip C# dll signing in ARM

* Copy ARM binaries to Nuget

* Restore nuget packages before ARM packaging

* wip

* Use host protoc at C# build

* Set ProtocDirectory on cross-compiled builds

* wip

* Fix typo
2020-02-10 16:29:27 -08:00
jignparm
645d8fb213
Jignparm/upgrade macos vm image (#2928)
* update MacOS image to 10.14

* Update to macos 10.14
2020-01-28 20:24:45 -08:00
Changming Sun
e0c9cdaa73
Fix the nuget pipelines (#2901) 2020-01-23 20:02:18 -08:00
KeDengMS
f9f25ec047
Fix spurious component detection warning (#2857)
Fix spurious component detection warning
Use component detection template for all pipelines
2020-01-18 20:10:35 -08:00
Changming Sun
c4e4abce73
Run static code analyzer on most of our code (#2817) 2020-01-10 22:17:17 -08:00
Changming Sun
48e042868f
Update test data (#2356) 2020-01-10 10:52:23 -08:00
Hariharan Seshadri
ebfcad1c90
Add script for release Nuget validation (#2719)
* Initial commit

* Nits

* Disable a test temporarily

* Change working directory

* Test

* Add download python step

* Test update

* More changes

* Fix space issue

* Fix

* Verify nuget signing

* Fix

* Spaces

* PR feedback

* Nit

* Fix

* Fix

* Remove temporary changes
2020-01-08 18:42:22 +05:30