mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-05-16 21:00:14 +00:00
## Description 1. Convert some git submodules to cmake external projects 2. Update nsync from [1.23.0](https://github.com/google/nsync/releases/tag/1.23.0) to [1.25.0](https://github.com/google/nsync/releases/tag/1.25.0) 3. Update re2 from 2021-06-01 to 2022-06-01 4. Update wil from an old commit to 1.0.220914.1 tag 5. Update gtest to a newer commit so that it can optionally leverage absl/re2 for parsing command line flags. The following git submodules are deleted: 1. FP16 2. safeint 3. XNNPACK 4. cxxopts 5. dlpack 7. flatbuffers 8. googlebenchmark 9. json 10. mimalloc 11. mp11 12. pthreadpool More will come. ## Motivation and Context There are 3 ways of integrating 3rd party C/C++ libraries into ONNX Runtime: 1. Install them to a system location, then use cmake's find_package module to locate them. 2. Use git submodules 6. Use cmake's external projects(externalproject_add). At first when this project was just started, we considered both option 2 and option 3. We preferred option 2 because: 1. It's easier to handle authentication. At first this project was not open source, and it had some other non-public dependencies. If we use git submodule, ADO will handle authentication smoothly. Otherwise we need to manually pass tokens around and be very careful on not exposing them in build logs. 2. At that time, cmake fetched dependencies after "cmake" finished generating vcprojects/makefiles. So it was very difficult to make cflags consistent. Since cmake 3.11, it has a new command: FetchContent, which fetches dependencies when it generates vcprojects/makefiles just before add_subdirectories, so the parent project's variables/settings can be easily passed to the child projects. And when the project went on, we had some new concerns: 1. As we started to have more and more EPs and build configs, the number of submodules grew quickly. For more developers, most ORT submodules are not relevant to them. They shouldn't need to download all of them. 2. It is impossible to let two different build configs use two different versions of the same dependency. For example, right now we have protobuf 3.18.3 in the submodules. Then every EP must use the same version. Whenever we have a need to upgrade protobuf, we need to coordinate across the whole team and many external developers. I can't manage it anymore. 3. Some projects want to manage the dependencies in a different way, either because of their preference or because of compliance requirements. For example, some Microsoft teams want to use vcpkg, but we don't want to force every user of onnxruntime using vcpkg. 7. Someone wants to dynamically link to protobuf, but our build script only does static link. 8. Hard to handle security vulnerabilities. For example, whenever protobuf has a security patch, we have a lot of things to do. But if we allowed people to build ORT with a different version of protobuf without changing ORT"s source code, the customer who build ORT from source will be able to act on such things in a quicker way. They will not need to wait ORT having a patch release. 9. Every time we do a release, github will also publish a source file zip file and a source file tarball for us. But they are not usable, because they miss submodules. ### New features After this change, users will be able to: 1. Build the dependencies in the way they want, then install them to somewhere(for example, /usr or a temp folder). 2. Or download the dependencies by using cmake commands from these dependencies official website 3. Similar to the above, but use your private mirrors to migrate supply chain risks. 4. Use different versions of the dependencies, as long as our source code is compatible with them. For example, you may use you can't use protobuf 3.20.x as they need code changes in ONNX Runtime. 6. Only download the things the current build needs. 10. Avoid building external dependencies again and again in every build. ### Breaking change The onnxruntime_PREFER_SYSTEM_LIB build option is removed you could think from now it is default ON. If you don't like the new behavior, you can set FETCHCONTENT_TRY_FIND_PACKAGE_MODE to NEVER. Besides, for who relied on the onnxruntime_PREFER_SYSTEM_LIB build option, please be aware that this PR will change find_package calls from Module mode to Config mode. For example, in the past if you have installed protobuf from apt-get from ubuntu 20.04's official repo, find_package can find it and use it. But after this PR, it won't. This is because that protobuf version provided by Ubuntu 20.04 is too old to support the "config mode". It can be resolved by getting a newer version of protobuf from somewhere.
176 lines
7.5 KiB
CMake
176 lines
7.5 KiB
CMake
# Protocol Buffers - Google's data interchange format
|
|
# Copyright 2008 Google Inc. All rights reserved.
|
|
# https://developers.google.com/protocol-buffers/
|
|
#
|
|
# Redistribution and use in source and binary forms, with or without
|
|
# modification, are permitted provided that the following conditions are
|
|
# met:
|
|
#
|
|
# * Redistributions of source code must retain the above copyright
|
|
# notice, this list of conditions and the following disclaimer.
|
|
# * Redistributions in binary form must reproduce the above
|
|
# copyright notice, this list of conditions and the following disclaimer
|
|
# in the documentation and/or other materials provided with the
|
|
# distribution.
|
|
# * Neither the name of Google Inc. nor the names of its
|
|
# contributors may be used to endorse or promote products derived from
|
|
# this software without specific prior written permission.
|
|
#
|
|
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
|
|
# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
|
|
# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
|
|
# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
|
|
# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
|
|
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
|
|
# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
|
|
# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
|
|
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
|
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
|
|
#Changelog:
|
|
#copied from https://github.com/protocolbuffers/protobuf/blob/master/cmake/protobuf-config.cmake.in
|
|
#sed -i 's/protobuf_generate/onnxruntime_protobuf_generate/g' protobuf-config.cmake.orig
|
|
#replace 'protobuf::protoc' with ${PROTOC_EXECUTABLE} and ${PROTOC_DEPS}
|
|
#remove OUTDIR
|
|
|
|
function(onnxruntime_protobuf_generate)
|
|
include(CMakeParseArguments)
|
|
if(EXISTS "${ONNX_CUSTOM_PROTOC_EXECUTABLE}")
|
|
set(PROTOC_EXECUTABLE ${ONNX_CUSTOM_PROTOC_EXECUTABLE})
|
|
else()
|
|
set(PROTOC_EXECUTABLE $<TARGET_FILE:protobuf::protoc>)
|
|
set(PROTOC_DEPS protobuf::protoc)
|
|
endif()
|
|
set(_options APPEND_PATH NO_SRC_INCLUDES)
|
|
set(_singleargs LANGUAGE OUT_VAR EXPORT_MACRO GEN_SRC_PREFIX GEN_SRC_SUB_DIR)
|
|
if(COMMAND target_sources)
|
|
list(APPEND _singleargs TARGET)
|
|
endif()
|
|
set(_multiargs PROTOS IMPORT_DIRS GENERATE_EXTENSIONS)
|
|
|
|
cmake_parse_arguments(onnxruntime_protobuf_generate "${_options}" "${_singleargs}" "${_multiargs}" "${ARGN}")
|
|
|
|
if(NOT onnxruntime_protobuf_generate_PROTOS AND NOT onnxruntime_protobuf_generate_TARGET)
|
|
message(SEND_ERROR "Error: onnxruntime_protobuf_generate called without any targets or source files")
|
|
return()
|
|
endif()
|
|
|
|
if(NOT onnxruntime_protobuf_generate_OUT_VAR AND NOT onnxruntime_protobuf_generate_TARGET)
|
|
message(SEND_ERROR "Error: onnxruntime_protobuf_generate called without a target or output variable")
|
|
return()
|
|
endif()
|
|
|
|
if(NOT onnxruntime_protobuf_generate_LANGUAGE)
|
|
set(onnxruntime_protobuf_generate_LANGUAGE cpp)
|
|
endif()
|
|
string(TOLOWER ${onnxruntime_protobuf_generate_LANGUAGE} onnxruntime_protobuf_generate_LANGUAGE)
|
|
|
|
if(onnxruntime_protobuf_generate_EXPORT_MACRO AND onnxruntime_protobuf_generate_LANGUAGE STREQUAL cpp)
|
|
set(_dll_export_decl "dllexport_decl=${onnxruntime_protobuf_generate_EXPORT_MACRO}:")
|
|
endif()
|
|
|
|
if(NOT onnxruntime_protobuf_generate_EXTENSIONS)
|
|
if(onnxruntime_protobuf_generate_LANGUAGE STREQUAL cpp)
|
|
set(onnxruntime_protobuf_generate_EXTENSIONS .pb.h .pb.cc)
|
|
elseif(onnxruntime_protobuf_generate_LANGUAGE STREQUAL python)
|
|
set(onnxruntime_protobuf_generate_EXTENSIONS _pb2.py)
|
|
else()
|
|
message(SEND_ERROR "Error: onnxruntime_protobuf_generate given unknown Language ${LANGUAGE}, please provide a value for GENERATE_EXTENSIONS")
|
|
return()
|
|
endif()
|
|
endif()
|
|
|
|
if(onnxruntime_protobuf_generate_TARGET)
|
|
get_target_property(_source_list ${onnxruntime_protobuf_generate_TARGET} SOURCES)
|
|
foreach(_file ${_source_list})
|
|
if(_file MATCHES "proto$")
|
|
list(APPEND onnxruntime_protobuf_generate_PROTOS ${_file})
|
|
endif()
|
|
endforeach()
|
|
endif()
|
|
|
|
if(NOT onnxruntime_protobuf_generate_PROTOS)
|
|
message(SEND_ERROR "Error: onnxruntime_protobuf_generate could not find any .proto files")
|
|
return()
|
|
endif()
|
|
|
|
if (NOT onnxruntime_protobuf_generate_NO_SRC_INCLUDES)
|
|
if(onnxruntime_protobuf_generate_APPEND_PATH)
|
|
# Create an include path for each file specified
|
|
foreach(_file ${onnxruntime_protobuf_generate_PROTOS})
|
|
get_filename_component(_abs_file ${_file} ABSOLUTE)
|
|
get_filename_component(_abs_path ${_abs_file} PATH)
|
|
list(FIND _protobuf_include_path ${_abs_path} _contains_already)
|
|
if(${_contains_already} EQUAL -1)
|
|
list(APPEND _protobuf_include_path -I ${_abs_path})
|
|
endif()
|
|
endforeach()
|
|
else()
|
|
set(_protobuf_include_path -I ${CMAKE_CURRENT_SOURCE_DIR})
|
|
endif()
|
|
endif()
|
|
|
|
foreach(DIR ${onnxruntime_protobuf_generate_IMPORT_DIRS})
|
|
get_filename_component(ABS_PATH ${DIR} ABSOLUTE)
|
|
list(FIND _protobuf_include_path ${ABS_PATH} _contains_already)
|
|
if(${_contains_already} EQUAL -1)
|
|
list(APPEND _protobuf_include_path -I ${ABS_PATH})
|
|
endif()
|
|
endforeach()
|
|
|
|
set(_generated_srcs_all)
|
|
|
|
if (onnxruntime_protobuf_generate_GEN_SRC_PREFIX)
|
|
set(_src_prefix "${onnxruntime_protobuf_generate_GEN_SRC_PREFIX}")
|
|
else()
|
|
set(_src_prefix "")
|
|
endif()
|
|
|
|
if (onnxruntime_protobuf_generate_GEN_SRC_SUB_DIR)
|
|
set(_src_sub_dir "${onnxruntime_protobuf_generate_GEN_SRC_SUB_DIR}")
|
|
if (NOT EXISTS ${_dll_export_decl}${CMAKE_CURRENT_BINARY_DIR}/${_src_sub_dir})
|
|
file(MAKE_DIRECTORY ${_dll_export_decl}${CMAKE_CURRENT_BINARY_DIR}/${_src_sub_dir})
|
|
endif()
|
|
else()
|
|
set(_src_sub_dir "")
|
|
endif()
|
|
|
|
foreach(_proto ${onnxruntime_protobuf_generate_PROTOS})
|
|
get_filename_component(_abs_file ${_proto} ABSOLUTE)
|
|
get_filename_component(_basename ${_proto} NAME_WE)
|
|
|
|
set(_generated_srcs)
|
|
foreach(_ext ${onnxruntime_protobuf_generate_EXTENSIONS})
|
|
list(APPEND _generated_srcs "${CMAKE_CURRENT_BINARY_DIR}/${_src_sub_dir}${_src_prefix}${_basename}${_ext}")
|
|
endforeach()
|
|
list(APPEND _generated_srcs_all ${_generated_srcs})
|
|
|
|
if (onnxruntime_USE_FULL_PROTOBUF)
|
|
add_custom_command(
|
|
OUTPUT ${_generated_srcs}
|
|
COMMAND ${PROTOC_EXECUTABLE}
|
|
ARGS --${onnxruntime_protobuf_generate_LANGUAGE}_out ${_dll_export_decl}${CMAKE_CURRENT_BINARY_DIR}/${_src_sub_dir} ${_protobuf_include_path} ${_abs_file}
|
|
DEPENDS ${_abs_file} ${PROTOC_DEPS}
|
|
COMMENT "Running ${onnxruntime_protobuf_generate_LANGUAGE} protocol buffer (full) compiler on ${_proto}"
|
|
VERBATIM )
|
|
else()
|
|
add_custom_command(
|
|
OUTPUT ${_generated_srcs}
|
|
COMMAND ${PROTOC_EXECUTABLE}
|
|
ARGS --${onnxruntime_protobuf_generate_LANGUAGE}_out lite:${_dll_export_decl}${CMAKE_CURRENT_BINARY_DIR}/${_src_sub_dir} ${_protobuf_include_path} ${_abs_file}
|
|
DEPENDS ${_abs_file} ${PROTOC_DEPS}
|
|
COMMENT "Running ${onnxruntime_protobuf_generate_LANGUAGE} protocol buffer compiler (lite) on ${_proto}"
|
|
VERBATIM )
|
|
endif()
|
|
endforeach()
|
|
|
|
set_source_files_properties(${_generated_srcs_all} PROPERTIES GENERATED TRUE)
|
|
if(onnxruntime_protobuf_generate_OUT_VAR)
|
|
set(${onnxruntime_protobuf_generate_OUT_VAR} ${_generated_srcs_all} PARENT_SCOPE)
|
|
endif()
|
|
if(onnxruntime_protobuf_generate_TARGET)
|
|
target_sources(${onnxruntime_protobuf_generate_TARGET} PRIVATE ${_generated_srcs_all})
|
|
endif()
|
|
|
|
endfunction()
|