saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-05-30 23:18:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/tools/ci_build/github/azure-pipelines/templates/c-api-cpu.yml
Changming Sun 04900f96c1
Improve dependency management (#13523) 
## Description
1. Convert some git submodules to cmake external projects
2. Update nsync from
[1.23.0](https://github.com/google/nsync/releases/tag/1.23.0) to
[1.25.0](https://github.com/google/nsync/releases/tag/1.25.0)
3. Update re2 from 2021-06-01 to 2022-06-01
4. Update wil from an old commit to 1.0.220914.1 tag
5. Update gtest to a newer commit so that it can optionally leverage
absl/re2 for parsing command line flags.

The following git submodules are deleted:

1. FP16
2. safeint
3. XNNPACK
4. cxxopts
5. dlpack
7. flatbuffers
8. googlebenchmark
9. json
10. mimalloc
11. mp11
12. pthreadpool

More will come.

## Motivation and Context
There are 3 ways of integrating 3rd party C/C++ libraries into ONNX
Runtime:
1. Install them to a system location, then use cmake's find_package
module to locate them.
2.  Use git submodules 
6.  Use cmake's external projects(externalproject_add). 

At first when this project was just started, we considered both option 2
and option 3. We preferred option 2 because:

1. It's easier to handle authentication. At first this project was not
open source, and it had some other non-public dependencies. If we use
git submodule, ADO will handle authentication smoothly. Otherwise we
need to manually pass tokens around and be very careful on not exposing
them in build logs.
2. At that time, cmake fetched dependencies after "cmake" finished
generating vcprojects/makefiles. So it was very difficult to make cflags
consistent. Since cmake 3.11, it has a new command: FetchContent, which
fetches dependencies when it generates vcprojects/makefiles just before
add_subdirectories, so the parent project's variables/settings can be
easily passed to the child projects.

And when the project went on,  we had some new concerns:
1. As we started to have more and more EPs and build configs, the number
of submodules grew quickly. For more developers, most ORT submodules are
not relevant to them. They shouldn't need to download all of them.
2. It is impossible to let two different build configs use two different
versions of the same dependency. For example, right now we have protobuf
3.18.3 in the submodules. Then every EP must use the same version.
Whenever we have a need to upgrade protobuf, we need to coordinate
across the whole team and many external developers. I can't manage it
anymore.
3. Some projects want to manage the dependencies in a different way,
either because of their preference or because of compliance
requirements. For example, some Microsoft teams want to use vcpkg, but
we don't want to force every user of onnxruntime using vcpkg.
7. Someone wants to dynamically link to protobuf, but our build script
only does static link.
8. Hard to handle security vulnerabilities. For example, whenever
protobuf has a security patch, we have a lot of things to do. But if we
allowed people to build ORT with a different version of protobuf without
changing ORT"s source code, the customer who build ORT from source will
be able to act on such things in a quicker way. They will not need to
wait ORT having a patch release.
9. Every time we do a release, github will also publish a source file
zip file and a source file tarball for us. But they are not usable,
because they miss submodules.
 
### New features

After this change, users will be able to:
1. Build the dependencies in the way they want, then install them to
somewhere(for example, /usr or a temp folder).
2. Or download the dependencies by using cmake commands from these
dependencies official website
3. Similar to the above, but use your private mirrors to migrate supply
chain risks.
4. Use different versions of the dependencies, as long as our source
code is compatible with them. For example, you may use you can't use
protobuf 3.20.x as they need code changes in ONNX Runtime.
6.  Only download the things the current build needs.
10. Avoid building external dependencies again and again in every build.

### Breaking change
The onnxruntime_PREFER_SYSTEM_LIB build option is removed you could think from now 
it is default ON. If you don't like the new behavior, you can set FETCHCONTENT_TRY_FIND_PACKAGE_MODE to NEVER.
Besides, for who relied on the onnxruntime_PREFER_SYSTEM_LIB build
option, please be aware that this PR will change find_package calls from
Module mode to Config mode. For example, in the past if you have
installed protobuf from apt-get from ubuntu 20.04's official repo,
find_package can find it and use it. But after this PR, it won't. This
is because that protobuf version provided by Ubuntu 20.04 is too old to
support the "config mode". It can be resolved by getting a newer version
of protobuf from somewhere.
2022-12-01 09:51:59 -08:00

917 lines
35 KiB
YAML
Raw Blame History

parameters:
- name: RunOnnxRuntimeTests
displayName: Run Tests?
type: boolean
default: true
- name: DoCompliance
displayName: Run Compliance Tasks?
type: boolean
default: true
- name: DoEsrp
displayName: Run code sign tasks? Must be true if you are doing an Onnx Runtime release.
type: boolean
default: false
- name: IsReleaseBuild
displayName: Is a release build? Set it to true if you are doing an Onnx Runtime release.
type: boolean
default: false
- name: AdditionalBuildFlags
displayName: Additional build flags for build.py
type: string
default: ''
- name: AdditionalWinBuildFlags
displayName: Additional build flags that just for Windows Builds
type: string
default: ''
- name: OrtNugetPackageId
displayName: Package name for nuget
type: string
default: 'Microsoft.ML.OnnxRuntime'
- name: BuildVariant
type: string
default: 'default'
jobs:
- template: c-api-linux-cpu.yml
parameters:
AdditionalBuildFlags: ${{ parameters.AdditionalBuildFlags }}
BaseImage: 'centos:7'
OnnxruntimeArch: 'x64'
OnnxruntimeCFlags: '-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fstack-clash-protection -fcf-protection -O3 -Wl,--strip-all'
OnnxruntimeCXXFlags: '-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fstack-clash-protection -fcf-protection -O3 -Wl,--strip-all'
OnnxruntimeNodejsBindingArch: 'x64'
PoolName: 'Linux-CPU'
- template: c-api-linux-cpu.yml
parameters:
AdditionalBuildFlags: ${{ parameters.AdditionalBuildFlags }}
BaseImage: 'arm64v8/centos:7'
OnnxruntimeArch: 'aarch64'
OnnxruntimeCFlags: '-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fstack-clash-protection -O3 -Wl,--strip-all'
OnnxruntimeCXXFlags: '-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -fstack-clash-protection -O3 -Wl,--strip-all'
OnnxruntimeNodejsBindingArch: 'arm64'
PoolName: 'aiinfra-linux-ARM64-CPU-2019'
- template: mac-ci.yml
parameters:
AllowReleasedOpsetOnly: 1
BuildForAllArchs: true
- template: android-java-api-aar.yml
parameters:
buildConfig: 'Release'
buildSettings: '$(Build.SourcesDirectory)/tools/ci_build/github/android/default_mobile_aar_build_settings.json'
includedOpsConfig: '$(Build.SourcesDirectory)/tools/ci_build/github/android/mobile_package.required_operators.config'
artifactName: 'onnxruntime-android-mobile-aar'
job_name_suffix: 'Mobile'
packageName: 'onnxruntime-mobile'
- template: android-java-api-aar.yml
parameters:
buildConfig: 'Release'
buildSettings: '$(Build.SourcesDirectory)/tools/ci_build/github/android/default_full_aar_build_settings.json'
artifactName: 'onnxruntime-android-full-aar'
job_name_suffix: 'Full'
publish_executables: '1'
- job: iOS_Full_xcframework
workspace:
clean: all
pool:
vmImage: 'macOS-11'
timeoutInMinutes: 300
steps:
- template: set-version-number-variables-step.yml
- script: |
/bin/bash $(Build.SourcesDirectory)/tools/ci_build/github/apple/build_host_protoc.sh \
$(Build.SourcesDirectory) \
$(Build.BinariesDirectory)/protobuf \
$(Build.BinariesDirectory)/protobuf_install
displayName: Build Host Protoc
- script: |
set -e -x
python3 tools/ci_build/github/apple/build_ios_framework.py \
--build_dir "$(Build.BinariesDirectory)/ios_framework" \
--path_to_protoc_exe $(Build.BinariesDirectory)/protobuf_install/bin/protoc \
tools/ci_build/github/apple/default_full_ios_framework_build_settings.json
mkdir $(Build.BinariesDirectory)/artifacts
mkdir -p $(Build.BinariesDirectory)/artifacts_staging/onnxruntime-ios-xcframework-$(OnnxRuntimeVersion)
cp -R $(Build.BinariesDirectory)/ios_framework/framework_out/onnxruntime.xcframework \
$(Build.BinariesDirectory)/artifacts_staging/onnxruntime-ios-xcframework-$(OnnxRuntimeVersion)
pushd $(Build.BinariesDirectory)/artifacts_staging
zip -vr $(Build.BinariesDirectory)/artifacts/onnxruntime_xcframework.zip \
onnxruntime-ios-xcframework-$(OnnxRuntimeVersion)
popd
displayName: "Build iOS xcframework"
- script: |
python3 tools/ci_build/github/apple/test_ios_packages.py \
--fail_if_cocoapods_missing \
--framework_info_file "$(Build.BinariesDirectory)/ios_framework/framework_info.json" \
--c_framework_dir "$(Build.BinariesDirectory)/ios_framework/framework_out" \
--variant Full
displayName: "Test iOS framework"
- task: PublishBuildArtifacts@1
inputs:
pathtoPublish: '$(Build.BinariesDirectory)/artifacts'
artifactName: 'onnxruntime-ios-full-xcframework'
- template: component-governance-component-detection-steps.yml
parameters :
condition : 'succeeded'
- template: win-ci.yml
parameters:
DoCompliance: ${{ parameters.DoCompliance }}
DoEsrp: ${{ parameters.DoEsrp }}
job_name_suffix: CPU_x86_${{ parameters.BuildVariant }}
EnvSetupScript: setup_env_x86.bat
buildArch: x86
msbuildPlatform: Win32
packageName: x86
buildparameter: ${{ parameters.AdditionalBuildFlags }} ${{ parameters.AdditionalWinBuildFlags}}
runTests: ${{ parameters.RunOnnxRuntimeTests }}
buildJava: false
buildNodejs: false
- template: win-ci.yml
parameters:
DoCompliance: ${{ parameters.DoCompliance }}
DoEsrp: ${{ parameters.DoEsrp }}
job_name_suffix: CPU_arm_${{ parameters.BuildVariant }}
EnvSetupScript: setup_env.bat
buildArch: x64
msbuildPlatform: arm
packageName: arm
buildparameter: --arm ${{ parameters.AdditionalBuildFlags }} ${{ parameters.AdditionalWinBuildFlags}} --path_to_protoc_exe $(Build.BinariesDirectory)\RelWithDebInfo\installed\bin\protoc.exe
runTests: false
buildJava: false
buildNodejs: false
- template: win-ci.yml
parameters:
DoCompliance: ${{ parameters.DoCompliance }}
DoEsrp: ${{ parameters.DoEsrp }}
job_name_suffix: CPU_arm64_${{ parameters.BuildVariant }}
EnvSetupScript: setup_env.bat
buildArch: x64
msbuildPlatform: arm64
packageName: arm64
buildparameter: --build_nodejs --arm64 ${{ parameters.AdditionalBuildFlags }} ${{ parameters.AdditionalWinBuildFlags}} --path_to_protoc_exe $(Build.BinariesDirectory)\RelWithDebInfo\installed\bin\protoc.exe
runTests: false
buildJava: false
buildNodejs: true
- template: win-ci.yml
parameters:
DoCompliance: ${{ parameters.DoCompliance }}
DoEsrp: ${{ parameters.DoEsrp }}
job_name_suffix: CPU_x64_${{ parameters.BuildVariant }}
EnvSetupScript: setup_env.bat
buildArch: x64
msbuildPlatform: x64
packageName: x64
buildparameter: --build_java --build_nodejs ${{ parameters.AdditionalBuildFlags }} ${{ parameters.AdditionalWinBuildFlags}}
runTests: ${{ parameters.RunOnnxRuntimeTests }}
buildJava: true
buildNodejs: true
- job: Jar_Packaging
workspace:
clean: all
pool: 'Win-CPU-2021'
dependsOn:
- Linux_C_API_Packaging_CPU_x64
- Linux_C_API_Packaging_CPU_aarch64
- MacOS_C_API_Packaging_CPU_x64
- Windows_Packaging_CPU_x86_${{ parameters.BuildVariant }}
- Windows_Packaging_CPU_x64_${{ parameters.BuildVariant }}
- Windows_Packaging_CPU_arm_${{ parameters.BuildVariant }}
- Windows_Packaging_CPU_arm64_${{ parameters.BuildVariant }}
condition: succeeded()
steps:
- checkout: self
submodules: false
- template: set-version-number-variables-step.yml
- task: DownloadPipelineArtifact@2
displayName: 'Download Pipeline Artifact - Win x64'
inputs:
buildType: 'current'
artifactName: 'drop-onnxruntime-java-win-x64'
targetPath: '$(Build.BinariesDirectory)\java-artifact\onnxruntime-java-win-x64'
- task: DownloadPipelineArtifact@2
displayName: 'Download Pipeline Artifact - Linux x64'
inputs:
buildType: 'current'
artifactName: 'drop-onnxruntime-java-linux-x64'
targetPath: '$(Build.BinariesDirectory)\java-artifact\onnxruntime-java-linux-x64'
- task: DownloadPipelineArtifact@2
displayName: 'Download Pipeline Artifact - Linux AARCH64'
inputs:
buildType: 'current'
artifactName: 'drop-onnxruntime-java-linux-aarch64'
targetPath: '$(Build.BinariesDirectory)\java-artifact\onnxruntime-java-linux-aarch64'
- task: DownloadPipelineArtifact@2
displayName: 'Download Pipeline Artifact - MacOS x64'
inputs:
buildType: 'current'
artifactName: 'drop-onnxruntime-java-osx-x86_64'
targetPath: '$(Build.BinariesDirectory)\java-artifact\onnxruntime-java-osx-x86_64'
- task: DownloadPipelineArtifact@2
displayName: 'Download Pipeline Artifact - MacOS ARM64'
inputs:
buildType: 'current'
artifactName: 'drop-onnxruntime-java-osx-arm64'
targetPath: '$(Build.BinariesDirectory)\java-artifact\onnxruntime-java-osx-arm64'
- task: PowerShell@2
displayName: 'PowerShell Script'
inputs:
targetType: filePath
filePath: $(Build.SourcesDirectory)\tools\ci_build\github\windows\jar_packaging.ps1
failOnStderr: true
showWarnings: true
workingDirectory: '$(Build.BinariesDirectory)\java-artifact'
- task: CopyFiles@2
displayName: 'Copy Java Files to Artifact Staging Directory'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\java-artifact\onnxruntime-java-win-x64'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishPipelineArtifact@1
displayName: 'Publish Pipeline Artifact'
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)'
artifact: 'onnxruntime-java'
- template: component-governance-component-detection-steps.yml
parameters :
condition : 'succeeded'
- task: mspremier.PostBuildCleanup.PostBuildCleanup-task.PostBuildCleanup@3
displayName: 'Clean Agent Directories'
condition: always()
- job: NuGet_Packaging_CPU
workspace:
clean: all
# we need to use the 2022 pool to create the nuget package with both pre-net6+Xamarin and net6 targets.
# VS2019 has no support for net6 and we need to use msbuild (from the VS install) to do the packing
pool: 'Azure-Pipelines-EO-Windows2022-aiinfra'
variables:
OrtPackageId: ${{ parameters.OrtNugetPackageId }}
breakCodesignValidationInjection: ${{ parameters.DoEsrp }}
dependsOn:
- Linux_C_API_Packaging_CPU_x64
- Linux_C_API_Packaging_CPU_aarch64
- MacOS_C_API_Packaging_CPU_x64
- Windows_Packaging_CPU_x86_${{ parameters.BuildVariant }}
- Windows_Packaging_CPU_x64_${{ parameters.BuildVariant }}
- Windows_Packaging_CPU_arm_${{ parameters.BuildVariant }}
- Windows_Packaging_CPU_arm64_${{ parameters.BuildVariant }}
- Android_Java_API_AAR_Packaging_Full
- iOS_Full_xcframework
condition: succeeded()
steps:
- checkout: self
submodules: true
- task: DownloadPipelineArtifact@0
displayName: 'Download win-x64 Pipeline Artifact'
inputs:
artifactName: 'onnxruntime-win-x64'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@0
displayName: 'Download win-x86 Pipeline Artifact'
inputs:
artifactName: 'onnxruntime-win-x86'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@0
displayName: 'Download win-arm64 Pipeline Artifact'
inputs:
artifactName: 'onnxruntime-win-arm64'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@0
displayName: 'Download win-arm Pipeline Artifact'
inputs:
artifactName: 'onnxruntime-win-arm'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@0
displayName: 'Download osx-x64 Pipeline Artifact'
inputs:
artifactName: 'onnxruntime-osx'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@0
displayName: 'Download linux-x64 Pipeline Artifact'
inputs:
artifactName: 'onnxruntime-linux-x64'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - NuGet'
inputs:
artifactName: 'onnxruntime-linux-aarch64'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@2
displayName: 'Download iOS Pipeline Artifact'
inputs:
artifactName: 'onnxruntime-ios-full-xcframework'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@2
displayName: 'Download android-full-aar Pipeline Artifact'
inputs:
artifactName: 'onnxruntime-android-full-aar'
patterns: '**/*.aar'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@0
displayName: 'Download drop-extra Pipeline Artifact'
inputs:
artifactName: 'drop-extra'
targetPath: '$(Build.BinariesDirectory)/extra-artifact'
- script: |
dir
workingDirectory: '$(Build.BinariesDirectory)/nuget-artifact'
displayName: 'List artifacts'
# Reconstruct the build dir
- task: PowerShell@2
displayName: 'Extract native libraries for addition to nuget native package'
inputs:
targetType: filePath
filePath: $(Build.SourcesDirectory)\tools\ci_build\github\windows\extract_nuget_files.ps1
- script: |
mklink /D /J models C:\local\models
workingDirectory: '$(Build.BinariesDirectory)'
displayName: 'Create models link'
- task: NuGetToolInstaller@0
displayName: Use Nuget 6.2.1
inputs:
versionSpec: 6.2.1
- task: PowerShell@2
displayName: Install .NET 6 workloads
inputs:
targetType: 'inline'
script: |
dotnet workload install android ios macos
workingDirectory: '$(Build.SourcesDirectory)\csharp'
- task: PowerShell@2
displayName: Build Microsoft.ML.OnnxRuntime .NET 6 targets using dotnet
inputs:
targetType: 'inline'
# we don't specify 'Any CPU' as the platform here because if we do it gets added to the output path
# e.g. csharp\src\Microsoft.ML.OnnxRuntime\bin\Any CPU\RelWithDebInfo\net6.0-ios\
# which is inconsistent with the msbuild output path for the pre-.net6 targets
# e.g. csharp\src\Microsoft.ML.OnnxRuntime\bin\RelWithDebInfo\monoandroid11.0
# and makes it harder to do the packing
#
# 'Any CPU' is the default (first 'mixed' platform specified in the csproj) so this should be fine.
script: |
dotnet build .\src\Microsoft.ML.OnnxRuntime\Microsoft.ML.OnnxRuntime.csproj -p:SelectedTargets=Net6 -p:Configuration=RelWithDebInfo -p:OnnxRuntimeBuildDirectory="$(Build.BinariesDirectory)" -p:OrtPackageId=$(OrtPackageId) -p:IsReleaseBuild=${{ parameters.IsReleaseBuild }}
workingDirectory: '$(Build.SourcesDirectory)\csharp'
- task: MSBuild@1
displayName: 'Restore NuGet Packages and create project.assets.json for pre-.net6 targets'
inputs:
solution: '$(Build.SourcesDirectory)\csharp\OnnxRuntime.CSharp.sln'
platform: 'Any CPU'
configuration: RelWithDebInfo
msbuildArguments: '-t:restore -p:SelectedTargets=PreNet6 -p:OrtPackageId=$(OrtPackageId)'
workingDirectory: '$(Build.SourcesDirectory)\csharp'
- task: MSBuild@1
displayName: 'Build C# for pre-.net6 targets'
inputs:
solution: '$(Build.SourcesDirectory)\csharp\OnnxRuntime.CSharp.sln'
platform: 'Any CPU'
configuration: RelWithDebInfo
msbuildArguments: '-p:SelectedTargets=PreNet6 -p:OnnxRuntimeBuildDirectory="$(Build.BinariesDirectory)" -p:OrtPackageId=$(OrtPackageId) -p:IsReleaseBuild=${{ parameters.IsReleaseBuild }}'
workingDirectory: '$(Build.SourcesDirectory)\csharp'
- template: win-esrp-dll.yml
parameters:
FolderPath: '$(Build.SourcesDirectory)\csharp\src\Microsoft.ML.OnnxRuntime\bin\RelWithDebInfo'
DisplayName: 'ESRP - Sign C# dlls'
DoEsrp: ${{ parameters.DoEsrp }}
- task: MSBuild@1
displayName: Update projects.assets.json with combined list of all target frameworks
inputs:
solution: '$(Build.SourcesDirectory)\csharp\src\Microsoft.ML.OnnxRuntime\Microsoft.ML.OnnxRuntime.csproj'
platform: 'Any CPU'
configuration: RelWithDebInfo
msbuildArguments: '-t:restore -p:SelectedTargets=All -p:OrtPackageId=$(OrtPackageId)'
workingDirectory: '$(Build.SourcesDirectory)\csharp'
- task: MSBuild@1
displayName: 'Build Nuget Packages'
inputs:
solution: '$(Build.SourcesDirectory)\csharp\OnnxRuntime.CSharp.proj'
platform: 'Any CPU'
configuration: RelWithDebInfo
msbuildArguments: '-t:CreatePackage -p:OnnxRuntimeBuildDirectory="$(Build.BinariesDirectory)" -p:OrtPackageId=$(OrtPackageId) -p:IsReleaseBuild=${{ parameters.IsReleaseBuild }}'
workingDirectory: '$(Build.SourcesDirectory)\csharp'
- task: CopyFiles@2
displayName: 'Copy native nuget package to: $(Build.ArtifactStagingDirectory)'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\RelWithDebInfo\RelWithDebInfo'
Contents: '*.nupkg'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: CopyFiles@2
displayName: 'Copy native nuget symbols package to: $(Build.ArtifactStagingDirectory)'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\RelWithDebInfo\RelWithDebInfo'
Contents: '*.snupkg'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: CopyFiles@2
displayName: 'Copy managed nuget package to: $(Build.ArtifactStagingDirectory)'
inputs:
SourceFolder: '$(Build.SourcesDirectory)\csharp\src\Microsoft.ML.OnnxRuntime\bin\RelWithDebInfo'
Contents: '*.nupkg'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- template: esrp_nuget.yml
parameters:
DisplayName: 'ESRP - sign NuGet package'
FolderPath: '$(Build.ArtifactStagingDirectory)'
DoEsrp: ${{ parameters.DoEsrp }}
- template: validate-package.yml
parameters:
PackageType: 'nuget'
PackagePath: '$(Build.ArtifactStagingDirectory)'
PackageName: 'Microsoft.ML.OnnxRuntime.*nupkg'
PlatformsSupported: 'win-x64,win-x86,linux-x64,linux-arm64,osx.10.14-x64'
VerifyNugetSigning: false
- task: PublishPipelineArtifact@0
displayName: 'Publish Pipeline NuGet Artifact'
inputs:
artifactName: 'drop-signed-nuget-CPU'
targetPath: '$(Build.ArtifactStagingDirectory)'
- task: MSBuild@1
displayName: 'Clean C#'
inputs:
solution: '$(Build.SourcesDirectory)\csharp\OnnxRuntime.CSharp.sln'
platform: 'Any CPU'
configuration: RelWithDebInfo
msbuildArguments: '-t:Clean -p:OnnxRuntimeBuildDirectory="$(Build.BinariesDirectory)" -p:OrtPackageId=$(OrtPackageId)'
workingDirectory: '$(Build.SourcesDirectory)\csharp'
- task: RoslynAnalyzers@2
displayName: 'Run Roslyn Analyzers'
inputs:
userProvideBuildInfo: msBuildInfo
msBuildCommandline: '"C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Current\Bin\msbuild.exe" $(Build.SourcesDirectory)\csharp\OnnxRuntime.CSharp.sln -p:configuration="RelWithDebInfo" -p:Platform="Any CPU" -p:OnnxRuntimeBuildDirectory="$(Build.BinariesDirectory)" -p:OrtPackageId=$(OrtPackageId)'
condition: and(succeeded(), eq('${{ parameters.DoCompliance }}', true))
- template: component-governance-component-detection-steps.yml
parameters :
condition : 'succeeded'
- task: mspremier.PostBuildCleanup.PostBuildCleanup-task.PostBuildCleanup@3
displayName: 'Clean Agent Directories'
condition: always()
- job: Nodejs_Packaging_CPU
workspace:
clean: all
pool: 'Win-CPU-2021'
variables:
${{ if eq(parameters.IsReleaseBuild, true) }}:
NpmPackagingMode: 'release'
${{ if not(eq(parameters.IsReleaseBuild, true)) }}:
NpmPackagingMode: 'dev'
breakCodesignValidationInjection: ${{ parameters.DoEsrp }}
dependsOn:
- Linux_C_API_Packaging_CPU_x64
- Linux_C_API_Packaging_CPU_aarch64
- MacOS_C_API_Packaging_CPU_x64
- Windows_Packaging_CPU_x64_${{ parameters.BuildVariant }}
- Windows_Packaging_CPU_arm64_${{ parameters.BuildVariant }}
condition: succeeded()
steps:
- checkout: self
submodules: true
- script: |
echo.>>.gitattributes
echo /js/** text=auto eol=lf>>.gitattributes
rd /s /q js
git checkout -- js/**
git checkout -- .gitattributes
workingDirectory: '$(Build.SourcesDirectory)'
displayName: 'Testing: force EOL to lf on windows for /js/**'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - NuGet (Win x64)'
inputs:
artifactName: 'onnxruntime-win-x64'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - NuGet (Win ARM64)'
inputs:
artifactName: 'onnxruntime-win-arm64'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - NuGet (OSX)'
inputs:
artifactName: 'onnxruntime-osx'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - NuGet (Linux x64)'
inputs:
artifactName: 'onnxruntime-linux-x64'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - NuGet (Linux aarch64)'
inputs:
artifactName: 'onnxruntime-linux-aarch64'
targetPath: '$(Build.BinariesDirectory)/nuget-artifact'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - Nodejs (Win x64)'
inputs:
artifactName: 'drop-onnxruntime-nodejs-win-x64'
targetPath: '$(Build.BinariesDirectory)/nodejs-artifacts/win32/x64/'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - Nodejs (Win ARM64)'
inputs:
artifactName: 'drop-onnxruntime-nodejs-win-arm64'
targetPath: '$(Build.BinariesDirectory)/nodejs-artifacts/win32/arm64/'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - Nodejs (macOS x86_64)'
inputs:
artifactName: 'drop-onnxruntime-nodejs-osx-x86_64'
targetPath: '$(Build.BinariesDirectory)/nodejs-artifacts/darwin/x64/'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - Nodejs (macOS arm64)'
inputs:
artifactName: 'drop-onnxruntime-nodejs-osx-arm64'
targetPath: '$(Build.BinariesDirectory)/nodejs-artifacts/darwin/arm64/'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - Nodejs (Linux x64)'
inputs:
artifactName: 'drop-onnxruntime-nodejs-linux-x64'
targetPath: '$(Build.BinariesDirectory)/nodejs-artifacts/linux/x64/'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - Nodejs (Linux aarch64)'
inputs:
artifactName: 'drop-onnxruntime-nodejs-linux-aarch64'
targetPath: '$(Build.BinariesDirectory)/nodejs-artifacts/linux/arm64/'
- task: DownloadPipelineArtifact@0
displayName: 'Download Pipeline Artifact - NuGet'
inputs:
artifactName: 'drop-extra'
targetPath: '$(Build.BinariesDirectory)/extra-artifact'
- task: PowerShell@2
displayName: 'PowerShell Script'
inputs:
targetType: filePath
filePath: $(Build.SourcesDirectory)\tools\ci_build\github\windows\extract_nuget_files.ps1
- script: |
dir
workingDirectory: '$(Build.BinariesDirectory)/nuget-artifact'
displayName: 'List artifacts'
- script: |
npm ci
workingDirectory: '$(Build.SourcesDirectory)/js'
displayName: 'Install NPM packages /js'
- script: |
npm ci
workingDirectory: '$(Build.SourcesDirectory)/js/common'
displayName: 'Install NPM packages /js/common'
- script: |
npm ci
workingDirectory: '$(Build.SourcesDirectory)/js/node'
displayName: 'Install NPM packages /js/node'
# Node.js binding win32/x64
- task: CopyFiles@2
displayName: 'Copy nuget binaries to: $(Build.SourcesDirectory)\js\node\bin\napi-v3\win32\x64\'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\RelWithDebInfo\RelWithDebInfo\nuget-artifacts\onnxruntime-win-x64\lib'
Contents: '*.dll'
TargetFolder: '$(Build.SourcesDirectory)\js\node\bin\napi-v3\win32\x64'
- task: CopyFiles@2
displayName: 'Copy nodejs binaries to: $(Build.SourcesDirectory)\js\node\bin\napi-v3\win32\x64\'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\nodejs-artifacts\win32\x64'
Contents: '*.node'
TargetFolder: '$(Build.SourcesDirectory)\js\node\bin\napi-v3\win32\x64'
# Node.js binding win32/arm64
- task: CopyFiles@2
displayName: 'Copy nuget binaries to: $(Build.SourcesDirectory)\js\node\bin\napi-v3\win32\arm64\'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\RelWithDebInfo\RelWithDebInfo\nuget-artifacts\onnxruntime-win-arm64\lib'
Contents: '*.dll'
TargetFolder: '$(Build.SourcesDirectory)\js\node\bin\napi-v3\win32\arm64'
- task: CopyFiles@2
displayName: 'Copy nodejs binaries to: $(Build.SourcesDirectory)\js\node\bin\napi-v3\win32\arm64\'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\nodejs-artifacts\win32\arm64'
Contents: '*.node'
TargetFolder: '$(Build.SourcesDirectory)\js\node\bin\napi-v3\win32\arm64'
# Node.js binding linux/x64
- task: CopyFiles@2
displayName: 'Copy nuget binaries to: $(Build.SourcesDirectory)\js\node\bin\napi-v3\linux\x64\'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\RelWithDebInfo\RelWithDebInfo\nuget-artifacts\onnxruntime-linux-x64\lib'
Contents: 'libonnxruntime.so.*'
TargetFolder: '$(Build.SourcesDirectory)\js\node\bin\napi-v3\linux\x64'
- task: CopyFiles@2
displayName: 'Copy nodejs binaries to: $(Build.SourcesDirectory)\js\node\bin\napi-v3\linux\x64\'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\nodejs-artifacts\linux\x64'
Contents: '*.node'
TargetFolder: '$(Build.SourcesDirectory)\js\node\bin\napi-v3\linux\x64'
# Node.js binding linux/arm64
- task: CopyFiles@2
displayName: 'Copy nuget binaries to: $(Build.SourcesDirectory)\js\node\bin\napi-v3\linux\arm64\'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\RelWithDebInfo\RelWithDebInfo\nuget-artifacts\onnxruntime-linux-aarch64\lib'
Contents: 'libonnxruntime.so.*'
TargetFolder: '$(Build.SourcesDirectory)\js\node\bin\napi-v3\linux\arm64'
- task: CopyFiles@2
displayName: 'Copy nodejs binaries to: $(Build.SourcesDirectory)\js\node\bin\napi-v3\linux\arm64\'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\nodejs-artifacts\linux\arm64'
Contents: '*.node'
TargetFolder: '$(Build.SourcesDirectory)\js\node\bin\napi-v3\linux\arm64'
# Node.js binding darwin/x64
- task: CopyFiles@2
displayName: 'Copy nuget binaries to: $(Build.SourcesDirectory)\js\node\bin\napi-v3\darwin\x64\'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\RelWithDebInfo\RelWithDebInfo\nuget-artifacts\onnxruntime-osx-x86_64\lib'
Contents: 'libonnxruntime.*.dylib'
TargetFolder: '$(Build.SourcesDirectory)\js\node\bin\napi-v3\darwin\x64'
- task: CopyFiles@2
displayName: 'Copy nodejs binaries to: $(Build.SourcesDirectory)\js\node\bin\napi-v3\darwin\x64\'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\nodejs-artifacts\darwin\x64'
Contents: '*.node'
TargetFolder: '$(Build.SourcesDirectory)\js\node\bin\napi-v3\darwin\x64'
# Node.js binding darwin/arm64
- task: CopyFiles@2
displayName: 'Copy nuget binaries to: $(Build.SourcesDirectory)\js\node\bin\napi-v3\darwin\arm64\'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\RelWithDebInfo\RelWithDebInfo\nuget-artifacts\onnxruntime-osx-arm64\lib'
Contents: 'libonnxruntime.*.dylib'
TargetFolder: '$(Build.SourcesDirectory)\js\node\bin\napi-v3\darwin\arm64'
- task: CopyFiles@2
displayName: 'Copy nodejs binaries to: $(Build.SourcesDirectory)\js\node\bin\napi-v3\darwin\arm64\'
inputs:
SourceFolder: '$(Build.BinariesDirectory)\nodejs-artifacts\darwin\arm64'
Contents: '*.node'
TargetFolder: '$(Build.SourcesDirectory)\js\node\bin\napi-v3\darwin\arm64'
- task: PowerShell@2
inputs:
filePath: '$(Build.SourcesDirectory)\tools\ci_build\github\js\pack-npm-packages.ps1'
arguments: '$(NpmPackagingMode) $(Build.SourcesDirectory) node'
workingDirectory: $(Build.BinariesDirectory)
errorActionPreference: stop
displayName: 'Pack NPM packages'
- task: CopyFiles@2
displayName: 'Copy onnxruntime-common package to staging directory'
inputs:
SourceFolder: '$(Build.SourcesDirectory)/js/common/'
Contents: 'onnxruntime-common-*.tgz'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: CopyFiles@2
displayName: 'Copy onnxruntime-node package to staging directory'
inputs:
SourceFolder: '$(Build.SourcesDirectory)/js/node/'
Contents: 'onnxruntime-node-*.tgz'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishPipelineArtifact@0
displayName: 'Publish NPM packages files'
inputs:
artifactName: 'NPM_packages'
targetPath: '$(Build.ArtifactStagingDirectory)'
- template: component-governance-component-detection-steps.yml
parameters :
condition : 'succeeded'
- task: mspremier.PostBuildCleanup.PostBuildCleanup-task.PostBuildCleanup@3
displayName: 'Clean Agent Directories'
condition: always()
- template: ../nuget/templates/test_win.yml
parameters:
AgentPool : 'Win-CPU-2021'
Skipx86Tests : false
NugetPackageName : 'Microsoft.ML.OnnxRuntime'
ArtifactSuffix: 'CPU'
- template: ../nuget/templates/test_linux.yml
parameters:
AgentPool : Linux-CPU
NugetPackageName : 'Microsoft.ML.OnnxRuntime'
ArtifactSuffix: 'CPU'
- template: ../nuget/templates/test_macos.yml
parameters:
AgentPool : macOS-11
ArtifactSuffix: 'CPU'
- template: ../nodejs/templates/test_win.yml
parameters:
AgentPool : 'Win-CPU-2021'
JobSuffix : 'Win_CPU_x64'
- template: ../nodejs/templates/test_linux.yml
parameters:
AgentPool : 'Linux-CPU'
JobSuffix : 'Linux_CPU_x64'
- template: ../nodejs/templates/test_macos.yml
parameters:
JobSuffix : 'macOS_CPU_x64'
- job: Final_Jar_Testing_Windows
workspace:
clean: all
pool: 'Win-CPU-2021'
timeoutInMinutes: 60
variables:
- name: runCodesignValidationInjection
value: false
dependsOn:
Jar_Packaging
steps:
- template: set-version-number-variables-step.yml
- task: DownloadPipelineArtifact@2
displayName: 'Download Final Jar'
inputs:
buildType: 'current'
artifactName: 'onnxruntime-java'
targetPath: '$(Build.BinariesDirectory)\final-jar'
- task: CmdLine@2
inputs:
script: |
mkdir test
pushd test
jar xf $(Build.BinariesDirectory)\final-jar\testing.jar
popd
powershell -Command "Invoke-WebRequest https://oss.sonatype.org/service/local/repositories/releases/content/org/junit/platform/junit-platform-console-standalone/1.6.2/junit-platform-console-standalone-1.6.2.jar -OutFile junit-platform-console-standalone-1.6.2.jar"
powershell -Command "Invoke-WebRequest https://oss.sonatype.org/service/local/repositories/releases/content/com/google/protobuf/protobuf-java/3.21.7/protobuf-java-3.21.7.jar -OutFile protobuf-java-3.21.7.jar"
java -jar junit-platform-console-standalone-1.6.2.jar -cp .;.\test;protobuf-java-3.21.7.jar;onnxruntime-$(OnnxRuntimeVersion).jar --scan-class-path --fail-if-no-tests --disable-banner
workingDirectory: '$(Build.BinariesDirectory)\final-jar'
- template: component-governance-component-detection-steps.yml
parameters :
condition : 'succeeded'
- task: mspremier.PostBuildCleanup.PostBuildCleanup-task.PostBuildCleanup@3
displayName: 'Clean Agent Directories'
condition: always()
- job: Final_Jar_Testing_Linux
workspace:
clean: all
pool: 'Linux-CPU'
variables:
- name: runCodesignValidationInjection
value: false
timeoutInMinutes: 60
dependsOn:
Jar_Packaging
steps:
- template: set-version-number-variables-step.yml
- task: DownloadPipelineArtifact@2
displayName: 'Download Final Jar'
inputs:
buildType: 'current'
artifactName: 'onnxruntime-java'
targetPath: '$(Build.BinariesDirectory)/final-jar'
- task: CmdLine@2
inputs:
script: |
echo "Java Version"
java --version
mkdir test
pushd test
jar xf $(Build.BinariesDirectory)/final-jar/testing.jar
popd
wget https://oss.sonatype.org/service/local/repositories/releases/content/org/junit/platform/junit-platform-console-standalone/1.6.2/junit-platform-console-standalone-1.6.2.jar -P ./
wget https://oss.sonatype.org/service/local/repositories/releases/content/com/google/protobuf/protobuf-java/3.21.7/protobuf-java-3.21.7.jar -P ./
LD_LIBRARY_PATH=./test:${LD_LIBRARY_PATH}
java -jar ./junit-platform-console-standalone-1.6.2.jar -cp .:./test:./protobuf-java-3.21.7.jar:./onnxruntime-$(OnnxRuntimeVersion).jar --scan-class-path --fail-if-no-tests --disable-banner
workingDirectory: '$(Build.BinariesDirectory)/final-jar'
- template: component-governance-component-detection-steps.yml
parameters :
condition : 'succeeded'
- task: mspremier.PostBuildCleanup.PostBuildCleanup-task.PostBuildCleanup@3
displayName: 'Clean Agent Directories'
condition: always()
- job: Final_Jar_Testing_MacOs
workspace:
clean: all
pool:
vmImage: 'macOS-11'
variables:
- name: runCodesignValidationInjection
value: false
timeoutInMinutes: 60
dependsOn:
Jar_Packaging
steps:
- template: set-version-number-variables-step.yml
- task: DownloadPipelineArtifact@2
displayName: 'Download Final Jar'
inputs:
buildType: 'current'
artifactName: 'onnxruntime-java'
targetPath: '$(Build.BinariesDirectory)/final-jar'
- task: CmdLine@2
inputs:
script: |
echo "Java Version"
java --version
mkdir test
pushd test
jar xf $(Build.BinariesDirectory)/final-jar/testing.jar
popd
wget https://oss.sonatype.org/service/local/repositories/releases/content/org/junit/platform/junit-platform-console-standalone/1.6.2/junit-platform-console-standalone-1.6.2.jar -P ./
wget https://oss.sonatype.org/service/local/repositories/releases/content/com/google/protobuf/protobuf-java/3.21.7/protobuf-java-3.21.7.jar -P ./
sudo xcode-select --switch /Applications/Xcode_12.4.app/Contents/Developer
DYLD_LIBRARY_PATH=./test:${DYLD_LIBRARY_PATH}
java -jar ./junit-platform-console-standalone-1.6.2.jar -cp .:./test:./protobuf-java-3.21.7.jar:./onnxruntime-$(OnnxRuntimeVersion).jar --scan-class-path --fail-if-no-tests --disable-banner
workingDirectory: '$(Build.BinariesDirectory)/final-jar'
- template: component-governance-component-detection-steps.yml
parameters :
condition : 'succeeded'
- task: mspremier.PostBuildCleanup.PostBuildCleanup-task.PostBuildCleanup@3
displayName: 'Clean Agent Directories'
condition: always()
- template: android-java-api-aar-test.yml
parameters:
artifactName: 'onnxruntime-android-mobile-aar'
job_name_suffix: 'Mobile'
packageName: 'onnxruntime-mobile'
- template: android-java-api-aar-test.yml
parameters:
artifactName: 'onnxruntime-android-full-aar'
job_name_suffix: 'Full'
Reference in a new issue View git blame Copy permalink