mirror of
https://github.com/saymrwulf/onnxruntime.git
synced 2026-07-29 20:14:01 +00:00
Remove orttraining/docker directory. (#3476)
The docker images are not publicly available yet. Addressing PR comment: https://github.com/microsoft/onnxruntime/pull/3174#discussion_r390761308
This commit is contained in:
edgchen1
GitHub
parent
03996c7c08
commit
20c7dd9f5c
5 changed files with 0 additions and 270 deletions
|
|
@ -1,6 +0,0 @@
|
|||
ARG BASE_IMAGE=onnxtraining.azurecr.io/azureml/base-gpu:openmpi4.0.0-cuda10.1-cudnn7-ubuntu16.04
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
WORKDIR /workspace
|
||||
|
||||
COPY onnxruntime_training_bert /workspace
|
||||
|
|
@ -1,109 +0,0 @@
|
|||
# AzureML Base GPU Docker Images
|
||||
AzureML provides base docker images for running on AzureML Compute (https://hub.docker.com/_/microsoft-azureml-base-gpu). However, these don't support CUDA 10.1 or OpenMPI 4.0. We provide Dockerfiles for these variants, as well as Dockerfiles to build and package onnxruntime training into an AzureML-compatible docker image.
|
||||
|
||||
## Prerequisites
|
||||
Install prerequisites.
|
||||
|
||||
1. Install Docker: https://docs.docker.com/install/
|
||||
1. Install Azure CLI: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest
|
||||
1. $ pip install azure-cli-core azure-mgmt-containerregistry azureml-sdk
|
||||
1. $ az login
|
||||
1. $ docker login onnxtraining.azurecr.io --username onnxtraining --password $(az acr credential show --name onnxtraining --subscription 'ea482afa-3a32-437c-aa10-7de928a9e793' --resource-group 'onnx_training' --query passwords[0].value)
|
||||
|
||||
## Build onnxruntime in an AzureML Docker container
|
||||
These docker images allow us to build onnxruntime with the desired library versions (CUDA/cuDNN/MPI/etc) and compatible with AzureML Compute.
|
||||
|
||||
Build onnxruntime using the desired docker image:
|
||||
```
|
||||
$ docker run --rm -v /path/to/onnxruntime:/onnxruntime onnxtraining.azurecr.io/azureml/build:openmpi3.1.2-cuda10.0-cudnn7-ubuntu16.04 <optional-build-arguments>
|
||||
$ docker run --rm -v /path/to/onnxruntime:/onnxruntime onnxtraining.azurecr.io/azureml/build:intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04 <optional-build-arguments>
|
||||
$ docker run --rm -v /path/to/onnxruntime:/onnxruntime onnxtraining.azurecr.io/azureml/build:openmpi4.0.0-cuda10.1-cudnn7-ubuntu16.04 <optional-build-arguments>
|
||||
$ docker run --rm -v /path/to/onnxruntime:/onnxruntime onnxtraining.azurecr.io/azureml/build:intelmpi2018.3-cuda10.1-cudnn7-ubuntu16.04 <optional-build-arguments>
|
||||
```
|
||||
|
||||
## Package onnxruntime into an AzureML Docker container
|
||||
Package the built onnxruntime_training_bert into an AzureML-compatible docker image (set the BASE_IMAGE corresponding to how onnxruntime was built above):
|
||||
|
||||
| Build image | BASE_IMAGE |
|
||||
| ----------- | ---------- |
|
||||
| onnxtraining.azurecr.io/azureml/build:openmpi3.1.2-cuda10.0-cudnn7-ubuntu16.04 | mcr.microsoft.com/azureml/base-gpu:openmpi3.1.2-cuda10.0-cudnn7-ubuntu16.04 |
|
||||
| onnxtraining.azurecr.io/azureml/build:intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04 | mcr.microsoft.com/azureml/base-gpu:intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04 |
|
||||
| onnxtraining.azurecr.io/azureml/build:openmpi4.0.0-cuda10.1-cudnn7-ubuntu16.04 | onnxtraining.azurecr.io/azureml/base-gpu:openmpi4.0.0-cuda10.1-cudnn7-ubuntu16.04 |
|
||||
| onnxtraining.azurecr.io/azureml/build:intelmpi2018.3-cuda10.1-cudnn7-ubuntu16.04 | onnxtraining.azurecr.io/azureml/base-gpu:intelmpi2018.3-cuda10.1-cudnn7-ubuntu16.04 |
|
||||
|
||||
```
|
||||
$ docker build --rm -t onnxtraining.azurecr.io/azureml/bert:<tag> --build-arg BASE_IMAGE=mcr.microsoft.com/azureml/base-gpu:openmpi3.1.2-cuda10.0-cudnn7-ubuntu16.04 -f ./Dockerfile build/Linux/RelWithDebInfo
|
||||
$ docker push onnxtraining.azurecr.io/azureml/bert:<tag>
|
||||
```
|
||||
|
||||
## Run the AureML Docker container on local compute
|
||||
The docker container can be run on a local machine. The model needs to be mounted into the container:
|
||||
```
|
||||
$ docker run --rm --runtime=nvidia -v /path/to/model:/model onnxtraining.azurecr.io/azureml/bert:<tag> /workspace/onnxruntime_training_bert --model_name /model/bert_L-24_H-1024_A-16_V_30528_optimized_layer_norm --train_batch_size 4 --mode perf --num_of_perf_samples 400
|
||||
```
|
||||
|
||||
## Run an Experiment on AzureML Compute
|
||||
After the AzureML-compatible docker container is created and pushed, an AzureML Experiment can be submitted to run on AzureML Compute. An Azure Portal link will be displayed to view progress and logs. Two AzureML Compute targets are pre-created for use:
|
||||
|
||||
1. 'onnx-training': 16-node NC24s-v3 cluster (use Open MPI docker images)
|
||||
1. 'intel-training': 16-node NC24rs-v3 cluster (use Intel MPI docker images)
|
||||
|
||||
```
|
||||
// Submit a single-node, single-GPU experiment (defaults to OpenMPI 4.0.0, CUDA 10.1, bert_L-24_H-1024_A-16_V_30528_optimized_layer_norm.onnx):
|
||||
$ python experiment.py --script_params='--train_batch_size=4 --mode=perf --num_of_perf_samples=400'
|
||||
Experiment running at: https://mlworkspace.azure.ai/portal/subscriptions/ea482afa-3a32-437c-aa10-7de928a9e793/resourceGroups/onnx_training/providers/Microsoft.MachineLearningServices/workspaces/ort_training_dev/experiments/BERT-ONNX
|
||||
|
||||
// Submit a single-node, single-GPU experiment with mixed-precision training on a custom docker image using IntelMPI:
|
||||
$ python experiment.py --container='onnxtraining.azurecr.io/azureml/bert:jesseb-cd29764cbe-intelmpi2018.3-cuda10.1-cudnn7-ubuntu16.04' --compute_target='intel-training' --script_params='--train_batch_size=8 --mode=perf --num_of_perf_samples=800 --use_mixed_precision=True'
|
||||
|
||||
// Submit a multi-node, multi-GPU experiment (4-node x 4-GPU) with gradient accumulation:
|
||||
$ python experiment.py --node_count=4 --gpu_count=4 --script_params='--train_batch_size=4 --mode=perf --num_of_perf_samples=6400 --gradient_accumulation_steps=16'
|
||||
|
||||
// See all options for running experiments:
|
||||
$ python experiment.py --help
|
||||
```
|
||||
|
||||
## Tensorboard for AzureML Experiments
|
||||
ONNX Training tensorboard events can be streamed to a local directory for interactive viewing (updates approximately every 5 seconds).
|
||||
|
||||
```
|
||||
// Submit an Experiment with tensorboard logging enabled (must not be --mode=perf):
|
||||
$ python experiment.py --script_params='--train_batch_size=4 --display_loss_steps=10 --log_dir=logs/tensorboard/'
|
||||
|
||||
// Stream the tensorboard logs locally, or download logs if Experiment is already complete:
|
||||
$ python watch_experiment.py --remote_dir='logs/tensorboard/' --local_dir='C:/tensorboard' --run='BERT-ONNX_1578424056_a4b83d75'
|
||||
|
||||
// Start Tensorboard locally:
|
||||
$ python -m tensorboard.main --logdir C:/tensorboard
|
||||
```
|
||||
|
||||
# Additional AzureML GPU Docker images
|
||||
AzureML base GPU docker images only support CUDA 9/10. We provide Dockerfiles for additional variants below.
|
||||
|
||||
## AzureML Docker image: CUDA 10.1, cuDNN 7.6.3, NCCL 2.4.8, OpenMPI 4.0.0
|
||||
This image is already built and pushed to onnxtraining.azurecr.io. To build the docker image from scratch:
|
||||
```
|
||||
$ docker build --rm -t onnxtraining.azurecr.io/azureml/base-gpu:openmpi4.0.0-cuda10.1-cudnn7-ubuntu16.04 openmpi
|
||||
$ docker push onnxtraining.azurecr.io/azureml/base-gpu:openmpi4.0.0-cuda10.1-cudnn7-ubuntu16.04
|
||||
```
|
||||
|
||||
## AzureML Docker image: CUDA 10.1, cuDNN 7.6.3, NCCL 2.4.8, IntelMPI 2018.3.222
|
||||
This image is already built and pushed to onnxtraining.azurecr.io. To build the docker image from scratch:
|
||||
```
|
||||
$ docker build --rm -t onnxtraining.azurecr.io/azureml/base-gpu:intelmpi2018.3-cuda10.1-cudnn7-ubuntu16.04 intelmpi
|
||||
$ docker push onnxtraining.azurecr.io/azureml/base-gpu:intelmpi2018.3-cuda10.1-cudnn7-ubuntu16.04
|
||||
```
|
||||
|
||||
## AzureML Docker image: Build environment
|
||||
These docker images allow us to build onnxruntime with the various versions of libraries and package into an AzureML-compatible docker container.
|
||||
|
||||
These images are already built and pushed to onnxtraining.azurecr.io. To build the docker images from scratch:
|
||||
```
|
||||
$ docker build --rm -t onnxtraining.azurecr.io/azureml/build:openmpi3.1.2-cuda10.0-cudnn7-ubuntu16.04 --build-arg BASE_IMAGE=mcr.microsoft.com/azureml/base-gpu:openmpi3.1.2-cuda10.0-cudnn7-ubuntu16.04 build
|
||||
|
||||
$ docker build --rm -t onnxtraining.azurecr.io/azureml/build:intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04 --build-arg BASE_IMAGE=mcr.microsoft.com/azureml/base-gpu:intelmpi2018.3-cuda10.0-cudnn7-ubuntu16.04 build
|
||||
|
||||
$ docker build --rm -t onnxtraining.azurecr.io/azureml/build:openmpi4.0.0-cuda10.1-cudnn7-ubuntu16.04 --build-arg BASE_IMAGE=onnxtraining.azurecr.io/azureml/base-gpu:openmpi4.0.0-cuda10.1-cudnn7-ubuntu16.04 build
|
||||
|
||||
$ docker build --rm -t onnxtraining.azurecr.io/azureml/build:intelmpi2018.3-cuda10.1-cudnn7-ubuntu16.04 --build-arg BASE_IMAGE=onnxtraining.azurecr.io/azureml/base-gpu:intelmpi2018.3-cuda10.1-cudnn7-ubuntu16.04 build
|
||||
```
|
||||
|
|
@ -1,63 +0,0 @@
|
|||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License.
|
||||
|
||||
# Tag: cuda:10.1-cudnn7-devel-ubuntu16.04
|
||||
# Env: CUDA_VERSION=10.1.243
|
||||
# Env: NCCL_VERSION=2.4.8
|
||||
# Env: CUDNN_VERSION=7.6.3.30
|
||||
# Ubuntu 16.04
|
||||
FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu16.04
|
||||
|
||||
USER root:root
|
||||
|
||||
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
|
||||
ENV DEBIAN_FRONTEND noninteractive
|
||||
ENV LD_LIBRARY_PATH "/usr/local/cuda/extras/CUPTI/lib64:${LD_LIBRARY_PATH}"
|
||||
|
||||
# Install Common Dependencies
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
# SSH and RDMA
|
||||
libmlx4-1 \
|
||||
libmlx5-1 \
|
||||
librdmacm1 \
|
||||
libibverbs1 \
|
||||
libmthca1 \
|
||||
libdapl2 \
|
||||
dapl2-utils \
|
||||
openssh-client \
|
||||
openssh-server \
|
||||
iproute2 && \
|
||||
# Others
|
||||
apt-get install -y \
|
||||
build-essential \
|
||||
bzip2 \
|
||||
git=1:2.7.4-0ubuntu1.7 \
|
||||
wget \
|
||||
cpio && \
|
||||
apt-get clean -y && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Conda Environment
|
||||
ENV MINICONDA_VERSION 4.5.11
|
||||
ENV PATH /opt/miniconda/bin:$PATH
|
||||
RUN wget -qO /tmp/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh && \
|
||||
bash /tmp/miniconda.sh -bf -p /opt/miniconda && \
|
||||
conda clean -ay && \
|
||||
rm -rf /opt/miniconda/pkgs && \
|
||||
rm /tmp/miniconda.sh && \
|
||||
find / -type d -name __pycache__ | xargs rm -rf
|
||||
|
||||
# Intel MPI installation
|
||||
ENV INTEL_MPI_VERSION 2018.3.222
|
||||
ENV PATH $PATH:/opt/intel/compilers_and_libraries/linux/mpi/bin64
|
||||
RUN cd /tmp && \
|
||||
wget -q "http://registrationcenter-download.intel.com/akdlm/irc_nas/tec/13063/l_mpi_${INTEL_MPI_VERSION}.tgz" && \
|
||||
tar zxvf l_mpi_${INTEL_MPI_VERSION}.tgz && \
|
||||
sed -i -e 's/^ACCEPT_EULA=decline/ACCEPT_EULA=accept/g' /tmp/l_mpi_${INTEL_MPI_VERSION}/silent.cfg && \
|
||||
cd /tmp/l_mpi_${INTEL_MPI_VERSION} && \
|
||||
./install.sh -s silent.cfg --arch=intel64 && \
|
||||
cd / && \
|
||||
rm -rf /tmp/l_mpi_${INTEL_MPI_VERSION}* && \
|
||||
rm -rf /opt/intel/compilers_and_libraries_${INTEL_MPI_VERSION}/linux/mpi/intel64/lib/debug* && \
|
||||
echo "source /opt/intel/compilers_and_libraries_${INTEL_MPI_VERSION}/linux/mpi/intel64/bin/mpivars.sh" >> ~/.bashrc
|
||||
|
|
@ -1,28 +0,0 @@
|
|||
# This docker file is for building with ib-enabled ompi libraries
|
||||
ARG OPENMPI_VERSIONBASE=3.1
|
||||
ARG OPENMPI_VERSIONMINOR=2
|
||||
ARG OPENMPI_VERSION=${OPENMPI_VERSIONBASE}.${OPENMPI_VERSIONMINOR}
|
||||
FROM onnxtraining.azurecr.io/azureml/base-gpu:openmpi${OPENMPI_VERSION}-cuda10.0-cudnn7-ubuntu16.04
|
||||
|
||||
# Set ARGs again since FROM resets all ARGS...
|
||||
ARG OPENMPI_VERSIONBASE=3.1
|
||||
ARG OPENMPI_VERSIONMINOR=2
|
||||
ARG OPENMPI_VERSION=${OPENMPI_VERSIONBASE}.${OPENMPI_VERSIONMINOR}
|
||||
|
||||
# IB libs
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends libmlx4-1 libmlx5-1 librdmacm1 libibverbs1 libmthca1 libdapl2 dapl2-utils openssh-client openssh-server iproute2
|
||||
RUN apt-get install -y --no-install-recommends libnuma-dev
|
||||
RUN apt-get install -y --no-install-recommends cpio libibverbs1 libmlx4-1 libmlx5-1 librdmacm1 libmthca1 libdapl2 dapl2-utils pciutils ibutils ibverbs-utils rdmacm-utils infiniband-diags perftest librdmacm-dev libibverbs-dev
|
||||
# Install ucx so that ompi can use IB directly
|
||||
RUN wget https://github.com/openucx/ucx/releases/download/v1.6.1-rc2/ucx-1.6.1.tar.gz && tar zxf ucx-1.6.1.tar.gz && cd ucx-1.6.1 && ./configure --prefix=/usr/local --enable-optimizations --disable-assertions --disable-params-check --enable-mt && make -j $(nproc --all) && make install
|
||||
# OMPI
|
||||
RUN wget https://download.open-mpi.org/release/open-mpi/v${OPENMPI_VERSIONBASE}/openmpi-${OPENMPI_VERSION}.tar.gz && tar zxf openmpi-${OPENMPI_VERSION}.tar.gz && cd openmpi-${OPENMPI_VERSION} && ./configure --with-ucx=/usr/local/ --enable-mca-no-build=btl-uct --prefix=/usr/local/openmpi-${OPENMPI_VERSION}/ && make -j $(nproc --all) && make -j"$(nproc)" install
|
||||
RUN ln -fs /usr/local/openmpi-${OPENMPI_VERSION} /usr/local/mpi
|
||||
# Test installation
|
||||
RUN test -f /usr/local/openmpi-${OPENMPI_VERSION}/bin/mpic++
|
||||
ENV LD_LIBRARY_PATH=/usr/local/openmpi-${OPENMPI_VERSION}/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH
|
||||
ENV PATH=/usr/local/openmpi-${OPENMPI_VERSION}/bin:$PATH
|
||||
|
||||
WORKDIR /workspace
|
||||
|
||||
COPY onnxruntime_training_bert /workspace
|
||||
|
|
@ -1,64 +0,0 @@
|
|||
# Copyright (c) Microsoft Corporation. All rights reserved.
|
||||
# Licensed under the MIT License.
|
||||
|
||||
# Tag: cuda:10.1-cudnn7-devel-ubuntu16.04
|
||||
# Env: CUDA_VERSION=10.1.243
|
||||
# Env: NCCL_VERSION=2.4.8
|
||||
# Env: CUDNN_VERSION=7.6.3.30
|
||||
# Ubuntu 16.04
|
||||
FROM nvidia/cuda:10.1-cudnn7-devel-ubuntu16.04
|
||||
|
||||
USER root:root
|
||||
|
||||
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
|
||||
ENV DEBIAN_FRONTEND noninteractive
|
||||
ENV LD_LIBRARY_PATH "/usr/local/cuda/extras/CUPTI/lib64:${LD_LIBRARY_PATH}"
|
||||
ENV NCCL_DEBUG=INFO
|
||||
ENV HOROVOD_GPU_ALLREDUCE=NCCL
|
||||
|
||||
# Install Common Dependencies
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
# SSH and RDMA
|
||||
libmlx4-1 \
|
||||
libmlx5-1 \
|
||||
librdmacm1 \
|
||||
libibverbs1 \
|
||||
libmthca1 \
|
||||
libdapl2 \
|
||||
dapl2-utils \
|
||||
openssh-client \
|
||||
openssh-server \
|
||||
iproute2 && \
|
||||
# Others
|
||||
apt-get install -y \
|
||||
build-essential \
|
||||
bzip2 \
|
||||
git=1:2.7.4-0ubuntu1.7 \
|
||||
wget \
|
||||
cpio && \
|
||||
apt-get clean -y && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Conda Environment
|
||||
ENV MINICONDA_VERSION 4.5.11
|
||||
ENV PATH /opt/miniconda/bin:$PATH
|
||||
RUN wget -qO /tmp/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-${MINICONDA_VERSION}-Linux-x86_64.sh && \
|
||||
bash /tmp/miniconda.sh -bf -p /opt/miniconda && \
|
||||
conda clean -ay && \
|
||||
rm -rf /opt/miniconda/pkgs && \
|
||||
rm /tmp/miniconda.sh && \
|
||||
find / -type d -name __pycache__ | xargs rm -rf
|
||||
|
||||
# Open-MPI installation
|
||||
ENV OPENMPI_VERSION 4.0.0
|
||||
RUN mkdir /tmp/openmpi && \
|
||||
cd /tmp/openmpi && \
|
||||
wget https://download.open-mpi.org/release/open-mpi/v4.0/openmpi-${OPENMPI_VERSION}.tar.gz && \
|
||||
tar zxf openmpi-${OPENMPI_VERSION}.tar.gz && \
|
||||
cd openmpi-${OPENMPI_VERSION} && \
|
||||
./configure --enable-orterun-prefix-by-default && \
|
||||
make -j $(nproc) all && \
|
||||
make install && \
|
||||
ldconfig && \
|
||||
rm -rf /tmp/openmpi
|
||||
Loading…
Reference in a new issue