Yi Zhang 0d1da41ca8 Fix docker image layer caching to avoid redundant docker building and transient connection exceptions. (#21612) ... ### Description Improve docker commands to make docker image layer caching works. It can make docker building faster and more stable. So far, A100 pool's system disk is too small to use docker cache. We won't use pipeline cache for docker image and remove some legacy code. ### Motivation and Context There are often an exception of ``` 64.58 + curl https://nodejs.org/dist/v18.17.1/node-v18.17.1-linux-x64.tar.gz -sSL --retry 5 --retry-delay 30 --create-dirs -o /tmp/src/node-v18.17.1-linux-x64.tar.gz --fail 286.4 curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2) ``` Because Onnxruntime pipeline have been sending too many requests to download Nodejs in docker building. Which is the major reason of pipeline failing now In fact, docker image layer caching never works. We can always see the scrips are still running ``` #9 [3/5] RUN cd /tmp/scripts && /tmp/scripts/install_centos.sh && /tmp/scripts/install_deps.sh && rm -rf /tmp/scripts #9 0.234 /bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) #9 0.235 /bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) #9 0.235 /tmp/scripts/install_centos.sh: line 1: !/bin/bash: No such file or directory #9 0.235 ++ '[' '!' -f /etc/yum.repos.d/microsoft-prod.repo ']' #9 0.236 +++ tr -dc 0-9. #9 0.236 +++ cut -d . -f1 #9 0.238 ++ os_major_version=8 .... #9 60.41 + curl https://nodejs.org/dist/v18.17.1/node-v18.17.1-linux-x64.tar.gz -sSL --retry 5 --retry-delay 30 --create-dirs -o /tmp/src/node-v18.17.1-linux-x64.tar.gz --fail #9 60.59 + return 0 ... ``` This PR is improving the docker command to make image layer caching work. Thus, CI won't send so many redundant request of downloading NodeJS. ``` #9 [2/5] ADD scripts /tmp/scripts #9 CACHED #10 [3/5] RUN cd /tmp/scripts && /tmp/scripts/install_centos.sh && /tmp/scripts/install_deps.sh && rm -rf /tmp/scripts #10 CACHED #11 [4/5] RUN adduser --uid 1000 onnxruntimedev #11 CACHED #12 [5/5] WORKDIR /home/onnxruntimedev #12 CACHED ``` ###Reference https://docs.docker.com/build/drivers/ --------- Co-authored-by: Yi Zhang <your@email.com>		2024-08-06 21:37:09 +08:00
..
github	Fix docker image layer caching to avoid redundant docker building and transient connection exceptions. (#21612)	2024-08-06 21:37:09 +08:00
requirements	Replace inline pip install with pip install from requirements*.txt (#21106)	2024-07-22 12:39:10 -07:00
__init__.py
amd_hipify.py	[ROCm] Add SkipGroupNorm for ROCm EP (#19303)	2024-02-21 11:08:48 +08:00
build.py	Update ruff and clang-format versions (#21479)	2024-07-24 11:50:11 -07:00
clean_docker_image_cache.py	Bump ruff to 0.3.2 and black to 24 (#19878)	2024-03-13 10:00:32 -07:00
compile_triton.py	[Better Engineering] Bump ruff to 0.0.278 and fix new lint errors (#16789)	2023-07-21 12:53:41 -07:00
coverage.py	Format all python files under onnxruntime with black and isort (#11324)	2022-04-26 09:35:16 -07:00
gen_def.py	Update ruff and clang-format versions (#21479)	2024-07-24 11:50:11 -07:00
get_docker_image.py	Fix docker image layer caching to avoid redundant docker building and transient connection exceptions. (#21612)	2024-08-06 21:37:09 +08:00
logger.py	Format all python files under onnxruntime with black and isort (#11324)	2022-04-26 09:35:16 -07:00
op_registration_utils.py	Bump ruff to 0.3.2 and black to 24 (#19878)	2024-03-13 10:00:32 -07:00
op_registration_validator.py	Bump ruff to 0.3.2 and black to 24 (#19878)	2024-03-13 10:00:32 -07:00
patch_manylinux.py	[Better Engineering] Bump ruff to 0.0.278 and fix new lint errors (#16789)	2023-07-21 12:53:41 -07:00
policheck_exclusions.xml	Exculde hipify option from policheck (#13431)	2022-10-25 16:35:16 +08:00
reduce_op_kernels.py	Update ruff and clang-format versions (#21479)	2024-07-24 11:50:11 -07:00
replace_urls_in_deps.py	Update ruff and clang-format versions (#21479)	2024-07-24 11:50:11 -07:00
set-trigger-rules.py	Separating all GPU stages into different Pipelines (#21521)	2024-07-26 14:54:45 -07:00
update_tsaoptions.py	Adopt linrtunner as the linting tool - take 2 (#15085)	2023-03-24 15:29:03 -07:00
upload_python_package_to_azure_storage.py	Update ruff and clang-format versions (#21479)	2024-07-24 11:50:11 -07:00