saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-07-27 20:02:15 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/tools/ci_build/github/azure-pipelines/linux-cpu-minimal-build-ci-pipeline.yml
Changming Sun 04900f96c1
Improve dependency management (#13523) 
## Description
1. Convert some git submodules to cmake external projects
2. Update nsync from
[1.23.0](https://github.com/google/nsync/releases/tag/1.23.0) to
[1.25.0](https://github.com/google/nsync/releases/tag/1.25.0)
3. Update re2 from 2021-06-01 to 2022-06-01
4. Update wil from an old commit to 1.0.220914.1 tag
5. Update gtest to a newer commit so that it can optionally leverage
absl/re2 for parsing command line flags.

The following git submodules are deleted:

1. FP16
2. safeint
3. XNNPACK
4. cxxopts
5. dlpack
7. flatbuffers
8. googlebenchmark
9. json
10. mimalloc
11. mp11
12. pthreadpool

More will come.

## Motivation and Context
There are 3 ways of integrating 3rd party C/C++ libraries into ONNX
Runtime:
1. Install them to a system location, then use cmake's find_package
module to locate them.
2.  Use git submodules 
6.  Use cmake's external projects(externalproject_add). 

At first when this project was just started, we considered both option 2
and option 3. We preferred option 2 because:

1. It's easier to handle authentication. At first this project was not
open source, and it had some other non-public dependencies. If we use
git submodule, ADO will handle authentication smoothly. Otherwise we
need to manually pass tokens around and be very careful on not exposing
them in build logs.
2. At that time, cmake fetched dependencies after "cmake" finished
generating vcprojects/makefiles. So it was very difficult to make cflags
consistent. Since cmake 3.11, it has a new command: FetchContent, which
fetches dependencies when it generates vcprojects/makefiles just before
add_subdirectories, so the parent project's variables/settings can be
easily passed to the child projects.

And when the project went on,  we had some new concerns:
1. As we started to have more and more EPs and build configs, the number
of submodules grew quickly. For more developers, most ORT submodules are
not relevant to them. They shouldn't need to download all of them.
2. It is impossible to let two different build configs use two different
versions of the same dependency. For example, right now we have protobuf
3.18.3 in the submodules. Then every EP must use the same version.
Whenever we have a need to upgrade protobuf, we need to coordinate
across the whole team and many external developers. I can't manage it
anymore.
3. Some projects want to manage the dependencies in a different way,
either because of their preference or because of compliance
requirements. For example, some Microsoft teams want to use vcpkg, but
we don't want to force every user of onnxruntime using vcpkg.
7. Someone wants to dynamically link to protobuf, but our build script
only does static link.
8. Hard to handle security vulnerabilities. For example, whenever
protobuf has a security patch, we have a lot of things to do. But if we
allowed people to build ORT with a different version of protobuf without
changing ORT"s source code, the customer who build ORT from source will
be able to act on such things in a quicker way. They will not need to
wait ORT having a patch release.
9. Every time we do a release, github will also publish a source file
zip file and a source file tarball for us. But they are not usable,
because they miss submodules.
 
### New features

After this change, users will be able to:
1. Build the dependencies in the way they want, then install them to
somewhere(for example, /usr or a temp folder).
2. Or download the dependencies by using cmake commands from these
dependencies official website
3. Similar to the above, but use your private mirrors to migrate supply
chain risks.
4. Use different versions of the dependencies, as long as our source
code is compatible with them. For example, you may use you can't use
protobuf 3.20.x as they need code changes in ONNX Runtime.
6.  Only download the things the current build needs.
10. Avoid building external dependencies again and again in every build.

### Breaking change
The onnxruntime_PREFER_SYSTEM_LIB build option is removed you could think from now 
it is default ON. If you don't like the new behavior, you can set FETCHCONTENT_TRY_FIND_PACKAGE_MODE to NEVER.
Besides, for who relied on the onnxruntime_PREFER_SYSTEM_LIB build
option, please be aware that this PR will change find_package calls from
Module mode to Config mode. For example, in the past if you have
installed protobuf from apt-get from ubuntu 20.04's official repo,
find_package can find it and use it. But after this PR, it won't. This
is because that protobuf version provided by Ubuntu 20.04 is too old to
support the "config mode". It can be resolved by getting a newer version
of protobuf from somewhere.
2022-12-01 09:51:59 -08:00

304 lines
13 KiB
YAML
Raw Blame History

# This CI has the following steps:
# 1. Build full ORT, install the full ORT python wheel and use it to generate ort format test models
# and include ops config file for step 3.
# 2. Build minimal ORT including all the kernels and disable exceptions.
# This step is build only to safe-guard the --disable_exceptions option.
# 3. Build minimal ORT include only the kernels using the include ops config file from step 1,
# and the models from <repo root>/onnxruntime/test/testdata/, run UT, and use onnx_test_runner to
# test the ort format models generated in step 1.
# Exceptions are enabled in this step to help debugging in case of CI failure.
# This step builds and tests ORT with (3a) and without (3b) type reduction enabled.
# 4. Build minimal ORT with type reduction from a globally allowed types list.
# This step uses a hard-coded list of types which may not include the types needed by the models
# in <repo root>/onnxruntime/test/testdata/, so the tests for those models are skipped.
# 5. Build extended minimal ORT and run tests.
# 6. Build with all optional features disabled and no kernels.
# 6a: regular build with python enabled checks that the exclusions don't break code paths in a full build.
# 6b: minimal build with exceptions and python disabled checks that the exclusions don't break code paths in a
# minimal build.
# 6c: extended minimal build with exceptions and python disabled checks that the exclusions don't break code paths
# in an extended minimal build.
# 7. Build extended minimal ORT with NNAPI, with exceptions/RTTI/ml_ops disabled, for Android(arm64-v8a),
# this safe-guards the extended minimal build with NNAPI EP.
resources:
repositories:
- repository: manylinux
type: Github
endpoint: Microsoft
name: pypa/manylinux
ref: 1a61614cabfd6b91c6afd6d9e172cc5b838c65fe
jobs:
- job: Linux_CPU_Minimal_Build_E2E
timeoutInMinutes: 120
workspace:
clean: all
pool: Linux-CPU-2019
variables:
test_data_directory: $(Build.SourcesDirectory)/.test_data
steps:
- checkout: self
clean: true
submodules: none
- template: templates/get-docker-image-steps.yml
parameters:
Dockerfile: tools/ci_build/github/linux/docker/Dockerfile.manylinux2014_cpu
Context: tools/ci_build/github/linux/docker
DockerBuildArgs: "--build-arg BUILD_UID=$( id -u )"
Repository: onnxruntimecpubuild
- task: CmdLine@2
displayName: Create test data directory
inputs:
script: |
# Create a folder for all test data
mkdir -p $(test_data_directory)
# create empty config used in some parts
touch $(test_data_directory)/include_no_operators.config
workingDirectory: $(Build.SourcesDirectory)
- task: CmdLine@2
displayName: 1. Build full onnxruntime and generate ORT format test files
inputs:
script: |
docker run --rm \
--volume $(Build.SourcesDirectory):/onnxruntime_src \
--volume $(Build.BinariesDirectory):/build \
--volume $(test_data_directory):/home/onnxruntimedev/.test_data \
-e ALLOW_RELEASED_ONNX_OPSET_ONLY=1 \
-e NIGHTLY_BUILD \
-e BUILD_BUILDNUMBER \
onnxruntimecpubuild \
/bin/bash /onnxruntime_src/tools/ci_build/github/linux/ort_minimal/build_full_ort_and_create_ort_files.sh \
/build/1
workingDirectory: $(Build.SourcesDirectory)
- task: CmdLine@2
displayName: 2. Build minimal onnxruntime [exceptions DISABLED, type reduction DISABLED, training ops ENABLED]
inputs:
script: |
# We will try to build minimal ORT with exceptions disabled and training ops enabled
# Only the building process is verified here, no test will be performed
docker run --rm \
--volume $(Build.SourcesDirectory):/onnxruntime_src \
--volume $(Build.BinariesDirectory):/build \
-e ALLOW_RELEASED_ONNX_OPSET_ONLY=1 \
-e NIGHTLY_BUILD \
-e BUILD_BUILDNUMBER \
onnxruntimecpubuild \
/opt/python/cp37-cp37m/bin/python3 /onnxruntime_src/tools/ci_build/build.py \
--build_dir /build/2 --cmake_generator Ninja \
--config Debug \
--skip_submodule_sync \
--build_shared_lib \
--parallel \
--skip_tests \
--minimal_build \
--disable_exceptions \
--enable_training_ops
workingDirectory: $(Build.SourcesDirectory)
- task: CmdLine@2
displayName: 3a. Build minimal onnxruntime [exceptions ENABLED, type reduction DISABLED, custom ops ENABLED] and run tests
inputs:
script: |
docker run --rm \
--volume $(Build.SourcesDirectory):/onnxruntime_src \
--volume $(Build.BinariesDirectory):/build \
--volume $(test_data_directory):/home/onnxruntimedev/.test_data \
-e ALLOW_RELEASED_ONNX_OPSET_ONLY=1 \
-e NIGHTLY_BUILD \
-e BUILD_BUILDNUMBER \
onnxruntimecpubuild \
/bin/bash /onnxruntime_src/tools/ci_build/github/linux/ort_minimal/build_minimal_ort_and_run_tests.sh \
--build-directory /build/3a \
--reduced-ops-config /home/onnxruntimedev/.test_data/required_ops.ort_models.config \
--enable-custom-ops
workingDirectory: $(Build.SourcesDirectory)
- task: CmdLine@2
displayName: 3b. Build minimal onnxruntime [exceptions ENABLED, type reduction ENABLED] and run tests
inputs:
script: |
docker run --rm \
--volume $(Build.SourcesDirectory):/onnxruntime_src \
--volume $(Build.BinariesDirectory):/build \
--volume $(test_data_directory):/home/onnxruntimedev/.test_data \
-e ALLOW_RELEASED_ONNX_OPSET_ONLY=1 \
-e NIGHTLY_BUILD \
-e BUILD_BUILDNUMBER \
onnxruntimecpubuild \
/bin/bash /onnxruntime_src/tools/ci_build/github/linux/ort_minimal/build_minimal_ort_and_run_tests.sh \
--build-directory /build/3b \
--reduced-ops-config /home/onnxruntimedev/.test_data/required_ops_and_types.ort_models.config \
--enable-type-reduction
workingDirectory: $(Build.SourcesDirectory)
- task: CmdLine@2
displayName: 4. Build minimal onnxruntime [exceptions ENABLED, type reduction ENABLED (globally allowed types)] and run tests
inputs:
script: |
printf "%s\n%s\n" \
"!globally_allowed_types;bool,float,int8_t,uint8_t" \
"!no_ops_specified_means_all_ops_are_required" \
> $(test_data_directory)/globally_allowed_types.config && \
docker run --rm \
--volume $(Build.SourcesDirectory):/onnxruntime_src \
--volume $(Build.BinariesDirectory):/build \
--volume $(test_data_directory):/home/onnxruntimedev/.test_data \
-e ALLOW_RELEASED_ONNX_OPSET_ONLY=1 \
-e NIGHTLY_BUILD \
-e BUILD_BUILDNUMBER \
onnxruntimecpubuild \
/bin/bash /onnxruntime_src/tools/ci_build/github/linux/ort_minimal/build_minimal_ort_and_run_tests.sh \
--build-directory /build/4 \
--reduced-ops-config /home/onnxruntimedev/.test_data/globally_allowed_types.config \
--enable-type-reduction \
--skip-model-tests
workingDirectory: $(Build.SourcesDirectory)
- task: CmdLine@2
displayName: 5. Build extended minimal onnxruntime and run tests
inputs:
script: |
docker run --rm \
--volume $(Build.SourcesDirectory):/onnxruntime_src \
--volume $(Build.BinariesDirectory):/build \
-e ALLOW_RELEASED_ONNX_OPSET_ONLY=1 \
-e NIGHTLY_BUILD \
-e BUILD_BUILDNUMBER \
onnxruntimecpubuild \
/opt/python/cp37-cp37m/bin/python3 /onnxruntime_src/tools/ci_build/build.py \
--build_dir /build/5 --cmake_generator Ninja \
--config Debug \
--skip_submodule_sync \
--build_shared_lib \
--parallel \
--minimal_build extended
workingDirectory: $(Build.SourcesDirectory)
- task: CmdLine@2
displayName: 6a. Regular build with python and all optional features disabled.
inputs:
script: |
docker run --rm \
--volume $(Build.SourcesDirectory):/onnxruntime_src \
--volume $(Build.BinariesDirectory):/build \
--volume $(test_data_directory):/home/onnxruntimedev/.test_data \
-e ALLOW_RELEASED_ONNX_OPSET_ONLY=1 \
-e NIGHTLY_BUILD \
-e BUILD_BUILDNUMBER \
onnxruntimecpubuild \
/opt/python/cp37-cp37m/bin/python3 /onnxruntime_src/tools/ci_build/build.py \
--build_dir /build/6a \
--cmake_generator Ninja \
--config MinSizeRel \
--skip_submodule_sync \
--build_shared_lib \
--build_wheel \
--parallel \
--skip_tests \
--disable_ml_ops \
--include_ops_by_config /home/onnxruntimedev/.test_data/include_no_operators.config \
--cmake_extra_defines onnxruntime_DISABLE_SPARSE_TENSORS=ON \
onnxruntime_DISABLE_OPTIONAL_TYPE=ON \
onnxruntime_BUILD_UNIT_TESTS=OFF
workingDirectory: $(Build.SourcesDirectory)
- task: CmdLine@2
displayName: 6b. Minimal build with all optional features disabled.
inputs:
script: |
docker run --rm \
--volume $(Build.SourcesDirectory):/onnxruntime_src \
--volume $(Build.BinariesDirectory):/build \
--volume $(test_data_directory):/home/onnxruntimedev/.test_data \
-e ALLOW_RELEASED_ONNX_OPSET_ONLY=1 \
-e NIGHTLY_BUILD \
-e BUILD_BUILDNUMBER \
onnxruntimecpubuild \
/opt/python/cp37-cp37m/bin/python3 /onnxruntime_src/tools/ci_build/build.py \
--build_dir /build/6b \
--cmake_generator Ninja \
--config MinSizeRel \
--skip_submodule_sync \
--build_shared_lib \
--parallel \
--minimal_build \
--disable_exceptions \
--disable_ml_ops \
--skip_tests \
--enable_reduced_operator_type_support \
--include_ops_by_config /home/onnxruntimedev/.test_data/include_no_operators.config \
--cmake_extra_defines onnxruntime_DISABLE_SPARSE_TENSORS=ON \
onnxruntime_DISABLE_OPTIONAL_TYPE=ON \
onnxruntime_BUILD_UNIT_TESTS=OFF
workingDirectory: $(Build.SourcesDirectory)
- task: CmdLine@2
displayName: 6c. Extended minimal build with all optional features disabled.
inputs:
script: |
docker run --rm \
--volume $(Build.SourcesDirectory):/onnxruntime_src \
--volume $(Build.BinariesDirectory):/build \
--volume $(test_data_directory):/home/onnxruntimedev/.test_data \
-e ALLOW_RELEASED_ONNX_OPSET_ONLY=1 \
-e NIGHTLY_BUILD \
-e BUILD_BUILDNUMBER \
onnxruntimecpubuild \
/opt/python/cp37-cp37m/bin/python3 /onnxruntime_src/tools/ci_build/build.py \
--build_dir /build/6c \
--cmake_generator Ninja \
--config MinSizeRel \
--skip_submodule_sync \
--build_shared_lib \
--parallel \
--minimal_build extended \
--disable_exceptions \
--disable_ml_ops \
--skip_tests \
--enable_reduced_operator_type_support \
--include_ops_by_config /home/onnxruntimedev/.test_data/include_no_operators.config \
--cmake_extra_defines onnxruntime_DISABLE_SPARSE_TENSORS=ON \
onnxruntime_DISABLE_OPTIONAL_TYPE=ON \
onnxruntime_BUILD_UNIT_TESTS=OFF
workingDirectory: $(Build.SourcesDirectory)
- task: CmdLine@2
displayName: 7. Extended minimal build with NNAPI EP for Android(arm64-v8a) and skip tests.
inputs:
script: |
NDK_HOME=$(realpath $ANDROID_NDK_HOME)
docker run --rm \
--volume $(Build.SourcesDirectory):/onnxruntime_src \
--volume $(Build.BinariesDirectory):/build \
--volume $ANDROID_HOME:/android_home \
--volume $NDK_HOME:/ndk_home \
-e ALLOW_RELEASED_ONNX_OPSET_ONLY=1 \
-e NIGHTLY_BUILD \
onnxruntimecpubuild \
/opt/python/cp37-cp37m/bin/python3 /onnxruntime_src/tools/ci_build/build.py \
--build_dir /build/7 \
--cmake_generator Ninja \
--config MinSizeRel \
--skip_submodule_sync \
--parallel \
--android \
--android_sdk_path /android_home \
--android_ndk_path /ndk_home \
--android_abi=arm64-v8a \
--android_api=29 \
--use_nnapi \
--minimal_build extended \
--build_shared_lib \
--disable_ml_ops \
--disable_exceptions \
--skip_tests --path_to_protoc_exe /usr/bin/protoc
workingDirectory: $(Build.SourcesDirectory)
- template: templates/clean-agent-build-directory-step.yml
Reference in a new issue View git blame Copy permalink