saymrwulf/onnxruntime
1
0
Fork 0
mirror of https://github.com/saymrwulf/onnxruntime.git synced 2026-05-17 21:10:43 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
onnxruntime/cmake/external/helper_functions.cmake
Changming Sun 04900f96c1
Improve dependency management (#13523) 
## Description
1. Convert some git submodules to cmake external projects
2. Update nsync from
[1.23.0](https://github.com/google/nsync/releases/tag/1.23.0) to
[1.25.0](https://github.com/google/nsync/releases/tag/1.25.0)
3. Update re2 from 2021-06-01 to 2022-06-01
4. Update wil from an old commit to 1.0.220914.1 tag
5. Update gtest to a newer commit so that it can optionally leverage
absl/re2 for parsing command line flags.

The following git submodules are deleted:

1. FP16
2. safeint
3. XNNPACK
4. cxxopts
5. dlpack
7. flatbuffers
8. googlebenchmark
9. json
10. mimalloc
11. mp11
12. pthreadpool

More will come.

## Motivation and Context
There are 3 ways of integrating 3rd party C/C++ libraries into ONNX
Runtime:
1. Install them to a system location, then use cmake's find_package
module to locate them.
2.  Use git submodules 
6.  Use cmake's external projects(externalproject_add). 

At first when this project was just started, we considered both option 2
and option 3. We preferred option 2 because:

1. It's easier to handle authentication. At first this project was not
open source, and it had some other non-public dependencies. If we use
git submodule, ADO will handle authentication smoothly. Otherwise we
need to manually pass tokens around and be very careful on not exposing
them in build logs.
2. At that time, cmake fetched dependencies after "cmake" finished
generating vcprojects/makefiles. So it was very difficult to make cflags
consistent. Since cmake 3.11, it has a new command: FetchContent, which
fetches dependencies when it generates vcprojects/makefiles just before
add_subdirectories, so the parent project's variables/settings can be
easily passed to the child projects.

And when the project went on,  we had some new concerns:
1. As we started to have more and more EPs and build configs, the number
of submodules grew quickly. For more developers, most ORT submodules are
not relevant to them. They shouldn't need to download all of them.
2. It is impossible to let two different build configs use two different
versions of the same dependency. For example, right now we have protobuf
3.18.3 in the submodules. Then every EP must use the same version.
Whenever we have a need to upgrade protobuf, we need to coordinate
across the whole team and many external developers. I can't manage it
anymore.
3. Some projects want to manage the dependencies in a different way,
either because of their preference or because of compliance
requirements. For example, some Microsoft teams want to use vcpkg, but
we don't want to force every user of onnxruntime using vcpkg.
7. Someone wants to dynamically link to protobuf, but our build script
only does static link.
8. Hard to handle security vulnerabilities. For example, whenever
protobuf has a security patch, we have a lot of things to do. But if we
allowed people to build ORT with a different version of protobuf without
changing ORT"s source code, the customer who build ORT from source will
be able to act on such things in a quicker way. They will not need to
wait ORT having a patch release.
9. Every time we do a release, github will also publish a source file
zip file and a source file tarball for us. But they are not usable,
because they miss submodules.
 
### New features

After this change, users will be able to:
1. Build the dependencies in the way they want, then install them to
somewhere(for example, /usr or a temp folder).
2. Or download the dependencies by using cmake commands from these
dependencies official website
3. Similar to the above, but use your private mirrors to migrate supply
chain risks.
4. Use different versions of the dependencies, as long as our source
code is compatible with them. For example, you may use you can't use
protobuf 3.20.x as they need code changes in ONNX Runtime.
6.  Only download the things the current build needs.
10. Avoid building external dependencies again and again in every build.

### Breaking change
The onnxruntime_PREFER_SYSTEM_LIB build option is removed you could think from now 
it is default ON. If you don't like the new behavior, you can set FETCHCONTENT_TRY_FIND_PACKAGE_MODE to NEVER.
Besides, for who relied on the onnxruntime_PREFER_SYSTEM_LIB build
option, please be aware that this PR will change find_package calls from
Module mode to Config mode. For example, in the past if you have
installed protobuf from apt-get from ubuntu 20.04's official repo,
find_package can find it and use it. But after this PR, it won't. This
is because that protobuf version provided by Ubuntu 20.04 is too old to
support the "config mode". It can be resolved by getting a newer version
of protobuf from somewhere.
2022-12-01 09:51:59 -08:00

199 lines
No EOL
8.9 KiB
CMake
Raw Blame History

# Distributed under the OSI-approved BSD 3-Clause License. See accompanying
# file Copyright.txt or https://cmake.org/licensing for details.
# This file was copied from cmake source with modifications:
# 1. Add the EXCLUDE_FROM_ALL keyword when this function calls add_subdirectory. It will also resolve the
# 'make install' issue.
# 2. Group the VC projects into the "external" folder. We can do it at there in a centralized way instead
# of doing it one by one.
# 3. Set the cmake property COMPILE_WARNING_AS_ERROR to OFF for these external projects.
macro(onnxruntime_fetchcontent_makeavailable)
# We must append an item, even if the variable is unset, so prefix its value.
# We will strip that prefix when we pop the value at the end of the macro.
list(APPEND __cmake_fcCurrentVarsStack
"__fcprefix__${CMAKE_VERIFY_INTERFACE_HEADER_SETS}"
)
set(CMAKE_VERIFY_INTERFACE_HEADER_SETS FALSE)
get_property(__cmake_providerCommand GLOBAL PROPERTY
__FETCHCONTENT_MAKEAVAILABLE_SERIAL_PROVIDER
)
foreach(__cmake_contentName IN ITEMS ${ARGV})
string(TOLOWER ${__cmake_contentName} __cmake_contentNameLower)
# If user specified FETCHCONTENT_SOURCE_DIR_... for this dependency, that
# overrides everything else and we shouldn't try to use find_package() or
# a dependency provider.
string(TOUPPER ${__cmake_contentName} __cmake_contentNameUpper)
if("${FETCHCONTENT_SOURCE_DIR_${__cmake_contentNameUpper}}" STREQUAL "")
# Dependency provider gets first opportunity, but prevent infinite
# recursion if we are called again for the same thing
if(NOT "${__cmake_providerCommand}" STREQUAL "" AND
NOT DEFINED __cmake_fcProvider_${__cmake_contentNameLower})
message(VERBOSE
"Trying FETCHCONTENT_MAKEAVAILABLE_SERIAL dependency provider for "
"${__cmake_contentName}"
)
# It's still valid if there are no saved details. The project may have
# been written to assume a dependency provider is always set and will
# provide dependencies without having any declared details for them.
__FetchContent_getSavedDetails(${__cmake_contentName} __cmake_contentDetails)
set(__cmake_providerArgs
"FETCHCONTENT_MAKEAVAILABLE_SERIAL"
"${__cmake_contentName}"
)
# Empty arguments must be preserved because of things like
# GIT_SUBMODULES (see CMP0097)
foreach(__cmake_item IN LISTS __cmake_contentDetails)
string(APPEND __cmake_providerArgs " [==[${__cmake_item}]==]")
endforeach()
# This property might be defined but empty. As long as it is defined,
# find_package() can be called.
get_property(__cmake_addfpargs GLOBAL PROPERTY
_FetchContent_${contentNameLower}_find_package_args
DEFINED
)
if(__cmake_addfpargs)
get_property(__cmake_fpargs GLOBAL PROPERTY
_FetchContent_${contentNameLower}_find_package_args
)
string(APPEND __cmake_providerArgs " FIND_PACKAGE_ARGS")
foreach(__cmake_item IN LISTS __cmake_fpargs)
string(APPEND __cmake_providerArgs " [==[${__cmake_item}]==]")
endforeach()
endif()
# Calling the provider could lead to onnxruntime_fetchcontent_makeavailable() being
# called for a nested dependency. That nested call may occur in the
# current variable scope. We have to save and restore the variables we
# need preserved.
list(APPEND __cmake_fcCurrentVarsStack
${__cmake_contentName}
${__cmake_contentNameLower}
)
set(__cmake_fcProvider_${__cmake_contentNameLower} YES)
cmake_language(EVAL CODE "${__cmake_providerCommand}(${__cmake_providerArgs})")
list(POP_BACK __cmake_fcCurrentVarsStack
__cmake_contentNameLower
__cmake_contentName
)
unset(__cmake_fcProvider_${__cmake_contentNameLower})
unset(__cmake_providerArgs)
unset(__cmake_addfpargs)
unset(__cmake_fpargs)
unset(__cmake_item)
unset(__cmake_contentDetails)
FetchContent_GetProperties(${__cmake_contentName})
if(${__cmake_contentNameLower}_POPULATED)
continue()
endif()
endif()
# Check if we've been asked to try find_package() first, even if we
# have already populated this dependency. If we previously tried to
# use find_package() for this and it succeeded, those things might
# no longer be in scope, so we have to do it again.
get_property(__cmake_haveFpArgs GLOBAL PROPERTY
_FetchContent_${__cmake_contentNameLower}_find_package_args DEFINED
)
if(__cmake_haveFpArgs)
unset(__cmake_haveFpArgs)
message(VERBOSE "Trying find_package(${__cmake_contentName} ...) before FetchContent")
get_property(__cmake_fpArgs GLOBAL PROPERTY
_FetchContent_${__cmake_contentNameLower}_find_package_args
)
# This call could lead to onnxruntime_fetchcontent_makeavailable() being called for
# a nested dependency and it may occur in the current variable scope.
# We have to save/restore the variables we need to preserve.
list(APPEND __cmake_fcCurrentNameStack
${__cmake_contentName}
${__cmake_contentNameLower}
)
find_package(${__cmake_contentName} ${__cmake_fpArgs})
list(POP_BACK __cmake_fcCurrentNameStack
__cmake_contentNameLower
__cmake_contentName
)
unset(__cmake_fpArgs)
if(${__cmake_contentName}_FOUND)
FetchContent_SetPopulated(${__cmake_contentName})
FetchContent_GetProperties(${__cmake_contentName})
continue()
endif()
endif()
else()
unset(__cmake_haveFpArgs)
endif()
FetchContent_GetProperties(${__cmake_contentName})
if(NOT ${__cmake_contentNameLower}_POPULATED)
FetchContent_Populate(${__cmake_contentName})
__FetchContent_setupFindPackageRedirection(${__cmake_contentName})
# Only try to call add_subdirectory() if the populated content
# can be treated that way. Protecting the call with the check
# allows this function to be used for projects that just want
# to ensure the content exists, such as to provide content at
# a known location. We check the saved details for an optional
# SOURCE_SUBDIR which can be used in the same way as its meaning
# for ExternalProject. It won't matter if it was passed through
# to the ExternalProject sub-build, since it would have been
# ignored there.
set(__cmake_srcdir "${${__cmake_contentNameLower}_SOURCE_DIR}")
__FetchContent_getSavedDetails(${__cmake_contentName} __cmake_contentDetails)
if("${__cmake_contentDetails}" STREQUAL "")
message(FATAL_ERROR "No details have been set for content: ${__cmake_contentName}")
endif()
cmake_parse_arguments(__cmake_arg "SYSTEM" "SOURCE_SUBDIR" "" ${__cmake_contentDetails})
if(NOT "${__cmake_arg_SOURCE_SUBDIR}" STREQUAL "")
string(APPEND __cmake_srcdir "/${__cmake_arg_SOURCE_SUBDIR}")
endif()
if(EXISTS ${__cmake_srcdir}/CMakeLists.txt)
add_subdirectory(${__cmake_srcdir} ${${__cmake_contentNameLower}_BINARY_DIR} EXCLUDE_FROM_ALL)
get_property(subdir_import_targets DIRECTORY "${__cmake_srcdir}" PROPERTY BUILDSYSTEM_TARGETS)
foreach(subdir_target ${subdir_import_targets})
if(TARGET ${subdir_target})
get_target_property(subdir_target_type ${subdir_target} TYPE)
if(subdir_target_type STREQUAL "EXECUTABLE")
get_target_property(subdir_target_osx_arch ${subdir_target} OSX_ARCHITECTURES)
if (subdir_target_osx_arch)
if (NOT ${CMAKE_HOST_SYSTEM_PROCESSOR} IN_LIST subdir_target_osx_arch)
message("Added an executable target ${subdir_target} but it can not run natively on ${CMAKE_HOST_SYSTEM_PROCESSOR}, we will try to modify it")
endif()
endif()
endif()
set_target_properties(${subdir_target} PROPERTIES FOLDER "External")
set_target_properties(${subdir_target} PROPERTIES COMPILE_WARNING_AS_ERROR OFF)
endif()
endforeach()
endif()
unset(__cmake_srcdir)
unset(__cmake_contentDetails)
unset(__cmake_arg_SOURCE_SUBDIR)
endif()
endforeach()
# Prefix will be "__fcprefix__"
list(POP_BACK __cmake_fcCurrentVarsStack __cmake_original_verify_setting)
string(SUBSTRING "${__cmake_original_verify_setting}"
12 -1 __cmake_original_verify_setting
)
set(CMAKE_VERIFY_INTERFACE_HEADER_SETS ${__cmake_original_verify_setting})
# clear local variables to prevent leaking into the caller's scope
unset(__cmake_contentName)
unset(__cmake_contentNameLower)
unset(__cmake_contentNameUpper)
unset(__cmake_providerCommand)
unset(__cmake_original_verify_setting)
endmacro()
Reference in a new issue View git blame Copy permalink