mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-05-16 21:00:14 +00:00
b854f2399d
1. Update manylinux build scripts. This will add [PEP600](https://www.python.org/dev/peps/pep-0600/)(manylinux2 tags) support. numpy has adopted this new feature, we should do the same. The old build script files were copied from https://github.com/pypa/manylinux, but they has been deleted and replaced in the upstream repo. The manylinux repo doesn't have a manylinux2014 branch anymore. So I'm removing the obsolete code, sync the files with the latest master. 2. Update GPU CUDA version from 11.0 to 11.1(after a discussion with PMs). 3. Delete tools/ci_build/github/linux/docker/Dockerfile.manylinux2014_cuda10_2. (Merged the content to tools/ci_build/github/linux/docker/Dockerfile.manylinux2014_cuda11) 4. Modernize the cmake code of how to locate python devel files. It was suggested in https://github.com/onnx/onnx/pull/1631 . 5. Remove `onnxruntime_MSVC_STATIC_RUNTIME` and `onnxruntime_GCC_STATIC_CPP_RUNTIME` build options. Now cmake has builtin support for it. Starting from cmake 3.15, we can use `CMAKE_MSVC_RUNTIME_LIBRARY` cmake variable to choose which MSVC runtime library we want to use. 6. Update Ubuntu docker images that used in our CI build from Ubuntu 18.04 to Ubuntu 20.04. 7. Update GCC version in CUDA 11.1 pipelines from 8.x to 9.3.1 8. Split Linux GPU CI pipeline to two jobs: build the code on a CPU machine then run the tests on another GPU machines. In the past we didn't test our python packages. We only tested the pre-packed files. So we didn't catch the rpath issue in CI build. 9. Add a CentOS machine pool and test our Linux GPU build on real CentOS machines. 10. Rework ARM64 Linux GPU python packaging pipeline. Previously it uses cross-compiling therefore we must static link to C Runtime. But now have pluggable EP API and it doesn't support static link. So I changed to use qemu emulation instead. Now the build is 10x slower than before. But it is more extensible.
71 lines
2.4 KiB
Bash
Executable file
71 lines
2.4 KiB
Bash
Executable file
#!/bin/bash
|
|
|
|
# Stop at any error, show all commands
|
|
set -exuo pipefail
|
|
|
|
# Get script directory
|
|
MY_DIR=$(dirname "${BASH_SOURCE[0]}")
|
|
|
|
# Get build utilities
|
|
source $MY_DIR/build_utils.sh
|
|
|
|
mkdir /opt/python
|
|
for PREFIX in $(find /opt/_internal/ -mindepth 1 -maxdepth 1 -name 'cpython*'); do
|
|
# Some python's install as bin/python3. Make them available as
|
|
# bin/python.
|
|
if [ -e ${PREFIX}/bin/python3 ] && [ ! -e ${PREFIX}/bin/python ]; then
|
|
ln -s python3 ${PREFIX}/bin/python
|
|
fi
|
|
${PREFIX}/bin/python -m ensurepip
|
|
if [ -e ${PREFIX}/bin/pip3 ] && [ ! -e ${PREFIX}/bin/pip ]; then
|
|
ln -s pip3 ${PREFIX}/bin/pip
|
|
fi
|
|
# Since we fall back on a canned copy of pip, we might not have
|
|
# the latest pip and friends. Upgrade them to make sure.
|
|
${PREFIX}/bin/pip install -U --require-hashes -r ${MY_DIR}/requirements.txt
|
|
# Create a symlink to PREFIX using the ABI_TAG in /opt/python/
|
|
ABI_TAG=$(${PREFIX}/bin/python ${MY_DIR}/python-tag-abi-tag.py)
|
|
ln -s ${PREFIX} /opt/python/${ABI_TAG}
|
|
# Make versioned python commands available directly in environment.
|
|
PYVERS=$(${PREFIX}/bin/python -c "import sys; print('.'.join(map(str, sys.version_info[:2])))")
|
|
ln -s ${PREFIX}/bin/python /usr/local/bin/python${PYVERS}
|
|
ln -s ${PREFIX}/bin/pip /usr/local/bin/pip${PYVERS}
|
|
done
|
|
|
|
# Create venv for auditwheel & certifi
|
|
TOOLS_PATH=/opt/_internal/tools
|
|
/opt/python/cp39-cp39/bin/python -m venv $TOOLS_PATH
|
|
source $TOOLS_PATH/bin/activate
|
|
|
|
# Install default packages
|
|
pip install -U --require-hashes -r $MY_DIR/requirements.txt
|
|
# Install certifi and auditwheel
|
|
pip install -U --require-hashes -r $MY_DIR/requirements-tools.txt
|
|
|
|
# Make auditwheel available in PATH
|
|
ln -s $TOOLS_PATH/bin/auditwheel /usr/local/bin/auditwheel
|
|
|
|
# Make pipx available in PATH
|
|
ln -s $TOOLS_PATH/bin/pipx /usr/local/bin/pipx
|
|
|
|
# Our openssl doesn't know how to find the system CA trust store
|
|
# (https://github.com/pypa/manylinux/issues/53)
|
|
# And it's not clear how up-to-date that is anyway
|
|
# So let's just use the same one pip and everyone uses
|
|
ln -s $(python -c 'import certifi; print(certifi.where())') /opt/_internal/certs.pem
|
|
# If you modify this line you also have to modify the versions in the Dockerfiles:
|
|
export SSL_CERT_FILE=/opt/_internal/certs.pem
|
|
|
|
# Deactivate the tools virtual environment
|
|
deactivate
|
|
|
|
# We do not need the precompiled .pyc and .pyo files.
|
|
clean_pyc /opt/_internal
|
|
|
|
# remove cache
|
|
rm -rf /root/.cache
|
|
|
|
hardlink -cv /opt/_internal
|
|
|
|
# update system packages
|
|
LC_ALL=C ${MY_DIR}/update-system-packages.sh
|