saymrwulf/pytorch
1
0
Fork 0
mirror of https://github.com/saymrwulf/pytorch.git synced 2026-05-14 20:57:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
pytorch/docker.Makefile

73 lines
2.4 KiB
Text
Raw Normal View History

docker: Add make variable to add docker build args (#48942) Summary: Adds an extra make variable 'EXTRA_DOCKER_BUILD_FLAGS' that allows us to add extra docker build flags to the docker build command. Example: make -f docker.Makefile EXTRA_DOCKER_BUILD_FLAGS=--no-cache devel-image Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/48942 Reviewed By: walterddr Differential Revision: D25376288 Pulled By: seemethere fbshipit-source-id: 9cf2c2a5e01d505fa54447604ecd653dcbdd42e1
2020-12-08 04:14:02 +00:00
DOCKER_REGISTRY = docker.io
DOCKER_ORG = $(shell docker info 2>/dev/null | sed '/Username:/!d;s/.* //')
DOCKER_IMAGE = pytorch
DOCKER_FULL_NAME = $(DOCKER_REGISTRY)/$(DOCKER_ORG)/$(DOCKER_IMAGE)
docker: Refactor Dockerfile process for official images (#32515) Summary: ## Commit Message: Refactors Dockerfile to be as parallel as possible with caching and adds a new Makefile to build said Dockerfile. Also updated the README.md to reflect the changes as well as updated some of the verbage around running our latest Docker images. Adds the new Dockerfile process to our CircleCI workflows ## How to build: Building the new images is pretty simple, just requires `docker` > 18.06 since the new build process relies on `buildkit` caching and multi-stage build resolving. ### Development images For `runtime` images: ``` make -f docker.Makefile runtime-image ``` For `devel` images: ``` make -f docker.Makefile devel-image ``` Builds are tagged as follows: ```bash docker.io/${docker_user:-whoami}/pytorch:$(git describe --tags)-${image_type} ``` Example: ``` docker.io/seemethere/pytorch:v1.4.0a0-2225-g9eba97b61d-runtime ``` ### Official images Official images are the ones hosted on [`docker.io/pytorch/pytorch`](https://hub.docker.com/r/pytorch/pytorch) To do official images builds you can simply add set the `BUILD_TYPE` variable to `official` and it will do the correct build without building the local binaries: Example: ``` make -f docker.Makefile BUILD_TYPE=official runtime-image ``` ## How to push: Pushing is also super simple (And will automatically tag the right thing based off of the git tag): ``` make -f docker.Makefile runtime-push make -f docker.Makefile devel-push ``` Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/32515 Differential Revision: D19558619 Pulled By: seemethere fbshipit-source-id: a06b25cd39ae9890751a60f8f36739ad6ab9ac99
2020-01-24 18:24:46 +00:00
ifeq ("$(DOCKER_ORG)","")
$(warning WARNING: No docker user found using results from whoami)
docker: Add make variable to add docker build args (#48942) Summary: Adds an extra make variable 'EXTRA_DOCKER_BUILD_FLAGS' that allows us to add extra docker build flags to the docker build command. Example: make -f docker.Makefile EXTRA_DOCKER_BUILD_FLAGS=--no-cache devel-image Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/48942 Reviewed By: walterddr Differential Revision: D25376288 Pulled By: seemethere fbshipit-source-id: 9cf2c2a5e01d505fa54447604ecd653dcbdd42e1
2020-12-08 04:14:02 +00:00
DOCKER_ORG = $(shell whoami)
docker: Refactor Dockerfile process for official images (#32515) Summary: ## Commit Message: Refactors Dockerfile to be as parallel as possible with caching and adds a new Makefile to build said Dockerfile. Also updated the README.md to reflect the changes as well as updated some of the verbage around running our latest Docker images. Adds the new Dockerfile process to our CircleCI workflows ## How to build: Building the new images is pretty simple, just requires `docker` > 18.06 since the new build process relies on `buildkit` caching and multi-stage build resolving. ### Development images For `runtime` images: ``` make -f docker.Makefile runtime-image ``` For `devel` images: ``` make -f docker.Makefile devel-image ``` Builds are tagged as follows: ```bash docker.io/${docker_user:-whoami}/pytorch:$(git describe --tags)-${image_type} ``` Example: ``` docker.io/seemethere/pytorch:v1.4.0a0-2225-g9eba97b61d-runtime ``` ### Official images Official images are the ones hosted on [`docker.io/pytorch/pytorch`](https://hub.docker.com/r/pytorch/pytorch) To do official images builds you can simply add set the `BUILD_TYPE` variable to `official` and it will do the correct build without building the local binaries: Example: ``` make -f docker.Makefile BUILD_TYPE=official runtime-image ``` ## How to push: Pushing is also super simple (And will automatically tag the right thing based off of the git tag): ``` make -f docker.Makefile runtime-push make -f docker.Makefile devel-push ``` Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/32515 Differential Revision: D19558619 Pulled By: seemethere fbshipit-source-id: a06b25cd39ae9890751a60f8f36739ad6ab9ac99
2020-01-24 18:24:46 +00:00
endif
docker: Update default cuda => 11.1 (#53299) Summary: We no longer build binaries for CUDA 11.0 so let's ensure that we have build for CUDA 11.1 by default instead Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/53299 Reviewed By: anjali411 Differential Revision: D26857194 Pulled By: seemethere fbshipit-source-id: 6094913922c0da832b96e5e49a67369d69d0b8ad
2021-03-05 22:41:43 +00:00
CUDA_VERSION = 11.1
docker: Add make variable to add docker build args (#48942) Summary: Adds an extra make variable 'EXTRA_DOCKER_BUILD_FLAGS' that allows us to add extra docker build flags to the docker build command. Example: make -f docker.Makefile EXTRA_DOCKER_BUILD_FLAGS=--no-cache devel-image Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/48942 Reviewed By: walterddr Differential Revision: D25376288 Pulled By: seemethere fbshipit-source-id: 9cf2c2a5e01d505fa54447604ecd653dcbdd42e1
2020-12-08 04:14:02 +00:00
CUDNN_VERSION = 8
BASE_RUNTIME = ubuntu:18.04
BASE_DEVEL = nvidia/cuda:$(CUDA_VERSION)-cudnn$(CUDNN_VERSION)-devel-ubuntu18.04
docker: Refactor Dockerfile process for official images (#32515) Summary: ## Commit Message: Refactors Dockerfile to be as parallel as possible with caching and adds a new Makefile to build said Dockerfile. Also updated the README.md to reflect the changes as well as updated some of the verbage around running our latest Docker images. Adds the new Dockerfile process to our CircleCI workflows ## How to build: Building the new images is pretty simple, just requires `docker` > 18.06 since the new build process relies on `buildkit` caching and multi-stage build resolving. ### Development images For `runtime` images: ``` make -f docker.Makefile runtime-image ``` For `devel` images: ``` make -f docker.Makefile devel-image ``` Builds are tagged as follows: ```bash docker.io/${docker_user:-whoami}/pytorch:$(git describe --tags)-${image_type} ``` Example: ``` docker.io/seemethere/pytorch:v1.4.0a0-2225-g9eba97b61d-runtime ``` ### Official images Official images are the ones hosted on [`docker.io/pytorch/pytorch`](https://hub.docker.com/r/pytorch/pytorch) To do official images builds you can simply add set the `BUILD_TYPE` variable to `official` and it will do the correct build without building the local binaries: Example: ``` make -f docker.Makefile BUILD_TYPE=official runtime-image ``` ## How to push: Pushing is also super simple (And will automatically tag the right thing based off of the git tag): ``` make -f docker.Makefile runtime-push make -f docker.Makefile devel-push ``` Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/32515 Differential Revision: D19558619 Pulled By: seemethere fbshipit-source-id: a06b25cd39ae9890751a60f8f36739ad6ab9ac99
2020-01-24 18:24:46 +00:00
Use CUDA 11.2 for nightly docker build. (#51990) Summary: Set CUDA_VERSION to 11.2.0 since Nvidia name their docker image on Ubuntu 18.04 to be nvidia/cuda:11.2.0-cudnn8-devel-ubuntu18.04. Note that cudatoolkit 11.2.0 is not yet on [conda](https://repo.anaconda.com/pkgs/main/linux-64/), and we need to wait for that before merging this PR. - https://hub.docker.com/r/nvidia/cuda/ Pull Request resolved: https://github.com/pytorch/pytorch/pull/51990 Reviewed By: samestep Differential Revision: D26371193 Pulled By: xuzhao9 fbshipit-source-id: 76915490dc30ddb03ceeeadb3c45a6c02b60401e
2021-02-10 18:43:22 +00:00
# The conda channel to use to install cudatoolkit
Use the conda channel defined in docker.Makefile to install cudatoolkit (#53316) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/53316 Test Plan: Nightly Docker build CI This is a follow-up PR after docker moved default CUDA => 11.1. Only merge this after https://github.com/pytorch/pytorch/issues/53299 is committed. Reviewed By: albanD Differential Revision: D26996287 Pulled By: xuzhao9 fbshipit-source-id: 0c2e03da41d036d7aada3e07d479a3dede219f58
2021-03-12 19:13:49 +00:00
CUDA_CHANNEL = nvidia
docker: Refactor Dockerfile process for official images (#32515) Summary: ## Commit Message: Refactors Dockerfile to be as parallel as possible with caching and adds a new Makefile to build said Dockerfile. Also updated the README.md to reflect the changes as well as updated some of the verbage around running our latest Docker images. Adds the new Dockerfile process to our CircleCI workflows ## How to build: Building the new images is pretty simple, just requires `docker` > 18.06 since the new build process relies on `buildkit` caching and multi-stage build resolving. ### Development images For `runtime` images: ``` make -f docker.Makefile runtime-image ``` For `devel` images: ``` make -f docker.Makefile devel-image ``` Builds are tagged as follows: ```bash docker.io/${docker_user:-whoami}/pytorch:$(git describe --tags)-${image_type} ``` Example: ``` docker.io/seemethere/pytorch:v1.4.0a0-2225-g9eba97b61d-runtime ``` ### Official images Official images are the ones hosted on [`docker.io/pytorch/pytorch`](https://hub.docker.com/r/pytorch/pytorch) To do official images builds you can simply add set the `BUILD_TYPE` variable to `official` and it will do the correct build without building the local binaries: Example: ``` make -f docker.Makefile BUILD_TYPE=official runtime-image ``` ## How to push: Pushing is also super simple (And will automatically tag the right thing based off of the git tag): ``` make -f docker.Makefile runtime-push make -f docker.Makefile devel-push ``` Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/32515 Differential Revision: D19558619 Pulled By: seemethere fbshipit-source-id: a06b25cd39ae9890751a60f8f36739ad6ab9ac99
2020-01-24 18:24:46 +00:00
# The conda channel to use to install pytorch / torchvision
docker: Add make variable to add docker build args (#48942) Summary: Adds an extra make variable 'EXTRA_DOCKER_BUILD_FLAGS' that allows us to add extra docker build flags to the docker build command. Example: make -f docker.Makefile EXTRA_DOCKER_BUILD_FLAGS=--no-cache devel-image Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/48942 Reviewed By: walterddr Differential Revision: D25376288 Pulled By: seemethere fbshipit-source-id: 9cf2c2a5e01d505fa54447604ecd653dcbdd42e1
2020-12-08 04:14:02 +00:00
INSTALL_CHANNEL = pytorch
docker: Refactor Dockerfile process for official images (#32515) Summary: ## Commit Message: Refactors Dockerfile to be as parallel as possible with caching and adds a new Makefile to build said Dockerfile. Also updated the README.md to reflect the changes as well as updated some of the verbage around running our latest Docker images. Adds the new Dockerfile process to our CircleCI workflows ## How to build: Building the new images is pretty simple, just requires `docker` > 18.06 since the new build process relies on `buildkit` caching and multi-stage build resolving. ### Development images For `runtime` images: ``` make -f docker.Makefile runtime-image ``` For `devel` images: ``` make -f docker.Makefile devel-image ``` Builds are tagged as follows: ```bash docker.io/${docker_user:-whoami}/pytorch:$(git describe --tags)-${image_type} ``` Example: ``` docker.io/seemethere/pytorch:v1.4.0a0-2225-g9eba97b61d-runtime ``` ### Official images Official images are the ones hosted on [`docker.io/pytorch/pytorch`](https://hub.docker.com/r/pytorch/pytorch) To do official images builds you can simply add set the `BUILD_TYPE` variable to `official` and it will do the correct build without building the local binaries: Example: ``` make -f docker.Makefile BUILD_TYPE=official runtime-image ``` ## How to push: Pushing is also super simple (And will automatically tag the right thing based off of the git tag): ``` make -f docker.Makefile runtime-push make -f docker.Makefile devel-push ``` Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/32515 Differential Revision: D19558619 Pulled By: seemethere fbshipit-source-id: a06b25cd39ae9890751a60f8f36739ad6ab9ac99
2020-01-24 18:24:46 +00:00
docker: Add make variable to add docker build args (#48942) Summary: Adds an extra make variable 'EXTRA_DOCKER_BUILD_FLAGS' that allows us to add extra docker build flags to the docker build command. Example: make -f docker.Makefile EXTRA_DOCKER_BUILD_FLAGS=--no-cache devel-image Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/48942 Reviewed By: walterddr Differential Revision: D25376288 Pulled By: seemethere fbshipit-source-id: 9cf2c2a5e01d505fa54447604ecd653dcbdd42e1
2020-12-08 04:14:02 +00:00
PYTHON_VERSION = 3.7
Add GH Actions CI to build nightly Docker and push to GitHub Container Registry (#51755) Summary: Currently PyTorch repository provides Dockerfile to build Docker with nightly builds, but it doesn't have CI to actually build those Dockers. This PR adds a GitHub action workflow to create PyTorch nightly build Docker and publish them to GitHub Container Registry. Also, add "--always" option to the `git describe --tags` command that generates the Docker image tag. Pull Request resolved: https://github.com/pytorch/pytorch/pull/51755 Test Plan: Manually trigger the workflow build in the GitHub Actions web UI. Reviewed By: seemethere Differential Revision: D26320180 Pulled By: xuzhao9 fbshipit-source-id: e00b472df14f5913cab9b06a41e837014e87f1c7
2021-02-08 22:56:45 +00:00
PYTORCH_VERSION = $(shell git describe --tags --always)
docker: Refactor Dockerfile process for official images (#32515) Summary: ## Commit Message: Refactors Dockerfile to be as parallel as possible with caching and adds a new Makefile to build said Dockerfile. Also updated the README.md to reflect the changes as well as updated some of the verbage around running our latest Docker images. Adds the new Dockerfile process to our CircleCI workflows ## How to build: Building the new images is pretty simple, just requires `docker` > 18.06 since the new build process relies on `buildkit` caching and multi-stage build resolving. ### Development images For `runtime` images: ``` make -f docker.Makefile runtime-image ``` For `devel` images: ``` make -f docker.Makefile devel-image ``` Builds are tagged as follows: ```bash docker.io/${docker_user:-whoami}/pytorch:$(git describe --tags)-${image_type} ``` Example: ``` docker.io/seemethere/pytorch:v1.4.0a0-2225-g9eba97b61d-runtime ``` ### Official images Official images are the ones hosted on [`docker.io/pytorch/pytorch`](https://hub.docker.com/r/pytorch/pytorch) To do official images builds you can simply add set the `BUILD_TYPE` variable to `official` and it will do the correct build without building the local binaries: Example: ``` make -f docker.Makefile BUILD_TYPE=official runtime-image ``` ## How to push: Pushing is also super simple (And will automatically tag the right thing based off of the git tag): ``` make -f docker.Makefile runtime-push make -f docker.Makefile devel-push ``` Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/32515 Differential Revision: D19558619 Pulled By: seemethere fbshipit-source-id: a06b25cd39ae9890751a60f8f36739ad6ab9ac99
2020-01-24 18:24:46 +00:00
# Can be either official / dev
docker: Add make variable to add docker build args (#48942) Summary: Adds an extra make variable 'EXTRA_DOCKER_BUILD_FLAGS' that allows us to add extra docker build flags to the docker build command. Example: make -f docker.Makefile EXTRA_DOCKER_BUILD_FLAGS=--no-cache devel-image Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/48942 Reviewed By: walterddr Differential Revision: D25376288 Pulled By: seemethere fbshipit-source-id: 9cf2c2a5e01d505fa54447604ecd653dcbdd42e1
2020-12-08 04:14:02 +00:00
BUILD_TYPE = dev
BUILD_PROGRESS = auto
BUILD_ARGS = --build-arg BASE_IMAGE=$(BASE_IMAGE) \
--build-arg PYTHON_VERSION=$(PYTHON_VERSION) \
--build-arg CUDA_VERSION=$(CUDA_VERSION) \
Use CUDA 11.2 for nightly docker build. (#51990) Summary: Set CUDA_VERSION to 11.2.0 since Nvidia name their docker image on Ubuntu 18.04 to be nvidia/cuda:11.2.0-cudnn8-devel-ubuntu18.04. Note that cudatoolkit 11.2.0 is not yet on [conda](https://repo.anaconda.com/pkgs/main/linux-64/), and we need to wait for that before merging this PR. - https://hub.docker.com/r/nvidia/cuda/ Pull Request resolved: https://github.com/pytorch/pytorch/pull/51990 Reviewed By: samestep Differential Revision: D26371193 Pulled By: xuzhao9 fbshipit-source-id: 76915490dc30ddb03ceeeadb3c45a6c02b60401e
2021-02-10 18:43:22 +00:00
--build-arg CUDA_CHANNEL=$(CUDA_CHANNEL) \
docker: add environment variable PYTORCH_VERSION (#50154) Summary: The aim is being able to inspect a container image and determine immediately which version of pytorch it contains. Closes https://github.com/pytorch/pytorch/issues/48324 Signed-off-by: Felix Abecassis <fabecassis@nvidia.com> seemethere PTAL. As you requested in https://github.com/pytorch/pytorch/issues/48324#issuecomment-754237156, I'm submitting the patch. But I could only do limited testing as I'm not sure these Makefile/Dockerfile are used for pushing the Docker Hub images (since the Makefile tags the image with a `v` prefix for the version, as in: `pytorch:v1.7.1`, but Docker Hub images don't have this prefix). Also on the master branch we currently have the following: ``` $ git describe --tags v1.4.0a0-11171-g68a6e46379 ``` So it's a little off, but it behaves as expected on the `release/1.7` branch. Pull Request resolved: https://github.com/pytorch/pytorch/pull/50154 Reviewed By: walterddr Differential Revision: D25828491 Pulled By: seemethere fbshipit-source-id: 500ec96cb5f5da1321610002d5e3678f4b0b94b5
2021-01-07 22:11:01 +00:00
--build-arg PYTORCH_VERSION=$(PYTORCH_VERSION) \
docker: Add make variable to add docker build args (#48942) Summary: Adds an extra make variable 'EXTRA_DOCKER_BUILD_FLAGS' that allows us to add extra docker build flags to the docker build command. Example: make -f docker.Makefile EXTRA_DOCKER_BUILD_FLAGS=--no-cache devel-image Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/48942 Reviewed By: walterddr Differential Revision: D25376288 Pulled By: seemethere fbshipit-source-id: 9cf2c2a5e01d505fa54447604ecd653dcbdd42e1
2020-12-08 04:14:02 +00:00
--build-arg INSTALL_CHANNEL=$(INSTALL_CHANNEL)
EXTRA_DOCKER_BUILD_FLAGS ?=
DOCKER_BUILD = DOCKER_BUILDKIT=1 \
docker build \
--progress=$(BUILD_PROGRESS) \
$(EXTRA_DOCKER_BUILD_FLAGS) \
--target $(BUILD_TYPE) \
-t $(DOCKER_FULL_NAME):$(DOCKER_TAG) \
$(BUILD_ARGS) .
DOCKER_PUSH = docker push $(DOCKER_FULL_NAME):$(DOCKER_TAG)
docker: Refactor Dockerfile process for official images (#32515) Summary: ## Commit Message: Refactors Dockerfile to be as parallel as possible with caching and adds a new Makefile to build said Dockerfile. Also updated the README.md to reflect the changes as well as updated some of the verbage around running our latest Docker images. Adds the new Dockerfile process to our CircleCI workflows ## How to build: Building the new images is pretty simple, just requires `docker` > 18.06 since the new build process relies on `buildkit` caching and multi-stage build resolving. ### Development images For `runtime` images: ``` make -f docker.Makefile runtime-image ``` For `devel` images: ``` make -f docker.Makefile devel-image ``` Builds are tagged as follows: ```bash docker.io/${docker_user:-whoami}/pytorch:$(git describe --tags)-${image_type} ``` Example: ``` docker.io/seemethere/pytorch:v1.4.0a0-2225-g9eba97b61d-runtime ``` ### Official images Official images are the ones hosted on [`docker.io/pytorch/pytorch`](https://hub.docker.com/r/pytorch/pytorch) To do official images builds you can simply add set the `BUILD_TYPE` variable to `official` and it will do the correct build without building the local binaries: Example: ``` make -f docker.Makefile BUILD_TYPE=official runtime-image ``` ## How to push: Pushing is also super simple (And will automatically tag the right thing based off of the git tag): ``` make -f docker.Makefile runtime-push make -f docker.Makefile devel-push ``` Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/32515 Differential Revision: D19558619 Pulled By: seemethere fbshipit-source-id: a06b25cd39ae9890751a60f8f36739ad6ab9ac99
2020-01-24 18:24:46 +00:00
.PHONY: all
all: devel-image
.PHONY: devel-image
devel-image: BASE_IMAGE := $(BASE_DEVEL)
docker: add environment variable PYTORCH_VERSION (#50154) Summary: The aim is being able to inspect a container image and determine immediately which version of pytorch it contains. Closes https://github.com/pytorch/pytorch/issues/48324 Signed-off-by: Felix Abecassis <fabecassis@nvidia.com> seemethere PTAL. As you requested in https://github.com/pytorch/pytorch/issues/48324#issuecomment-754237156, I'm submitting the patch. But I could only do limited testing as I'm not sure these Makefile/Dockerfile are used for pushing the Docker Hub images (since the Makefile tags the image with a `v` prefix for the version, as in: `pytorch:v1.7.1`, but Docker Hub images don't have this prefix). Also on the master branch we currently have the following: ``` $ git describe --tags v1.4.0a0-11171-g68a6e46379 ``` So it's a little off, but it behaves as expected on the `release/1.7` branch. Pull Request resolved: https://github.com/pytorch/pytorch/pull/50154 Reviewed By: walterddr Differential Revision: D25828491 Pulled By: seemethere fbshipit-source-id: 500ec96cb5f5da1321610002d5e3678f4b0b94b5
2021-01-07 22:11:01 +00:00
devel-image: DOCKER_TAG := $(PYTORCH_VERSION)-devel
docker: Refactor Dockerfile process for official images (#32515) Summary: ## Commit Message: Refactors Dockerfile to be as parallel as possible with caching and adds a new Makefile to build said Dockerfile. Also updated the README.md to reflect the changes as well as updated some of the verbage around running our latest Docker images. Adds the new Dockerfile process to our CircleCI workflows ## How to build: Building the new images is pretty simple, just requires `docker` > 18.06 since the new build process relies on `buildkit` caching and multi-stage build resolving. ### Development images For `runtime` images: ``` make -f docker.Makefile runtime-image ``` For `devel` images: ``` make -f docker.Makefile devel-image ``` Builds are tagged as follows: ```bash docker.io/${docker_user:-whoami}/pytorch:$(git describe --tags)-${image_type} ``` Example: ``` docker.io/seemethere/pytorch:v1.4.0a0-2225-g9eba97b61d-runtime ``` ### Official images Official images are the ones hosted on [`docker.io/pytorch/pytorch`](https://hub.docker.com/r/pytorch/pytorch) To do official images builds you can simply add set the `BUILD_TYPE` variable to `official` and it will do the correct build without building the local binaries: Example: ``` make -f docker.Makefile BUILD_TYPE=official runtime-image ``` ## How to push: Pushing is also super simple (And will automatically tag the right thing based off of the git tag): ``` make -f docker.Makefile runtime-push make -f docker.Makefile devel-push ``` Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/32515 Differential Revision: D19558619 Pulled By: seemethere fbshipit-source-id: a06b25cd39ae9890751a60f8f36739ad6ab9ac99
2020-01-24 18:24:46 +00:00
devel-image:
$(DOCKER_BUILD)
.PHONY: devel-image
devel-push: BASE_IMAGE := $(BASE_DEVEL)
docker: add environment variable PYTORCH_VERSION (#50154) Summary: The aim is being able to inspect a container image and determine immediately which version of pytorch it contains. Closes https://github.com/pytorch/pytorch/issues/48324 Signed-off-by: Felix Abecassis <fabecassis@nvidia.com> seemethere PTAL. As you requested in https://github.com/pytorch/pytorch/issues/48324#issuecomment-754237156, I'm submitting the patch. But I could only do limited testing as I'm not sure these Makefile/Dockerfile are used for pushing the Docker Hub images (since the Makefile tags the image with a `v` prefix for the version, as in: `pytorch:v1.7.1`, but Docker Hub images don't have this prefix). Also on the master branch we currently have the following: ``` $ git describe --tags v1.4.0a0-11171-g68a6e46379 ``` So it's a little off, but it behaves as expected on the `release/1.7` branch. Pull Request resolved: https://github.com/pytorch/pytorch/pull/50154 Reviewed By: walterddr Differential Revision: D25828491 Pulled By: seemethere fbshipit-source-id: 500ec96cb5f5da1321610002d5e3678f4b0b94b5
2021-01-07 22:11:01 +00:00
devel-push: DOCKER_TAG := $(PYTORCH_VERSION)-devel
docker: Refactor Dockerfile process for official images (#32515) Summary: ## Commit Message: Refactors Dockerfile to be as parallel as possible with caching and adds a new Makefile to build said Dockerfile. Also updated the README.md to reflect the changes as well as updated some of the verbage around running our latest Docker images. Adds the new Dockerfile process to our CircleCI workflows ## How to build: Building the new images is pretty simple, just requires `docker` > 18.06 since the new build process relies on `buildkit` caching and multi-stage build resolving. ### Development images For `runtime` images: ``` make -f docker.Makefile runtime-image ``` For `devel` images: ``` make -f docker.Makefile devel-image ``` Builds are tagged as follows: ```bash docker.io/${docker_user:-whoami}/pytorch:$(git describe --tags)-${image_type} ``` Example: ``` docker.io/seemethere/pytorch:v1.4.0a0-2225-g9eba97b61d-runtime ``` ### Official images Official images are the ones hosted on [`docker.io/pytorch/pytorch`](https://hub.docker.com/r/pytorch/pytorch) To do official images builds you can simply add set the `BUILD_TYPE` variable to `official` and it will do the correct build without building the local binaries: Example: ``` make -f docker.Makefile BUILD_TYPE=official runtime-image ``` ## How to push: Pushing is also super simple (And will automatically tag the right thing based off of the git tag): ``` make -f docker.Makefile runtime-push make -f docker.Makefile devel-push ``` Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/32515 Differential Revision: D19558619 Pulled By: seemethere fbshipit-source-id: a06b25cd39ae9890751a60f8f36739ad6ab9ac99
2020-01-24 18:24:46 +00:00
devel-push:
$(DOCKER_PUSH)
.PHONY: runtime-image
runtime-image: BASE_IMAGE := $(BASE_RUNTIME)
docker: add environment variable PYTORCH_VERSION (#50154) Summary: The aim is being able to inspect a container image and determine immediately which version of pytorch it contains. Closes https://github.com/pytorch/pytorch/issues/48324 Signed-off-by: Felix Abecassis <fabecassis@nvidia.com> seemethere PTAL. As you requested in https://github.com/pytorch/pytorch/issues/48324#issuecomment-754237156, I'm submitting the patch. But I could only do limited testing as I'm not sure these Makefile/Dockerfile are used for pushing the Docker Hub images (since the Makefile tags the image with a `v` prefix for the version, as in: `pytorch:v1.7.1`, but Docker Hub images don't have this prefix). Also on the master branch we currently have the following: ``` $ git describe --tags v1.4.0a0-11171-g68a6e46379 ``` So it's a little off, but it behaves as expected on the `release/1.7` branch. Pull Request resolved: https://github.com/pytorch/pytorch/pull/50154 Reviewed By: walterddr Differential Revision: D25828491 Pulled By: seemethere fbshipit-source-id: 500ec96cb5f5da1321610002d5e3678f4b0b94b5
2021-01-07 22:11:01 +00:00
runtime-image: DOCKER_TAG := $(PYTORCH_VERSION)-runtime
docker: Refactor Dockerfile process for official images (#32515) Summary: ## Commit Message: Refactors Dockerfile to be as parallel as possible with caching and adds a new Makefile to build said Dockerfile. Also updated the README.md to reflect the changes as well as updated some of the verbage around running our latest Docker images. Adds the new Dockerfile process to our CircleCI workflows ## How to build: Building the new images is pretty simple, just requires `docker` > 18.06 since the new build process relies on `buildkit` caching and multi-stage build resolving. ### Development images For `runtime` images: ``` make -f docker.Makefile runtime-image ``` For `devel` images: ``` make -f docker.Makefile devel-image ``` Builds are tagged as follows: ```bash docker.io/${docker_user:-whoami}/pytorch:$(git describe --tags)-${image_type} ``` Example: ``` docker.io/seemethere/pytorch:v1.4.0a0-2225-g9eba97b61d-runtime ``` ### Official images Official images are the ones hosted on [`docker.io/pytorch/pytorch`](https://hub.docker.com/r/pytorch/pytorch) To do official images builds you can simply add set the `BUILD_TYPE` variable to `official` and it will do the correct build without building the local binaries: Example: ``` make -f docker.Makefile BUILD_TYPE=official runtime-image ``` ## How to push: Pushing is also super simple (And will automatically tag the right thing based off of the git tag): ``` make -f docker.Makefile runtime-push make -f docker.Makefile devel-push ``` Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/32515 Differential Revision: D19558619 Pulled By: seemethere fbshipit-source-id: a06b25cd39ae9890751a60f8f36739ad6ab9ac99
2020-01-24 18:24:46 +00:00
runtime-image:
$(DOCKER_BUILD)
docker tag $(DOCKER_FULL_NAME):$(DOCKER_TAG) $(DOCKER_FULL_NAME):latest
.PHONY: runtime-image
runtime-push: BASE_IMAGE := $(BASE_RUNTIME)
docker: add environment variable PYTORCH_VERSION (#50154) Summary: The aim is being able to inspect a container image and determine immediately which version of pytorch it contains. Closes https://github.com/pytorch/pytorch/issues/48324 Signed-off-by: Felix Abecassis <fabecassis@nvidia.com> seemethere PTAL. As you requested in https://github.com/pytorch/pytorch/issues/48324#issuecomment-754237156, I'm submitting the patch. But I could only do limited testing as I'm not sure these Makefile/Dockerfile are used for pushing the Docker Hub images (since the Makefile tags the image with a `v` prefix for the version, as in: `pytorch:v1.7.1`, but Docker Hub images don't have this prefix). Also on the master branch we currently have the following: ``` $ git describe --tags v1.4.0a0-11171-g68a6e46379 ``` So it's a little off, but it behaves as expected on the `release/1.7` branch. Pull Request resolved: https://github.com/pytorch/pytorch/pull/50154 Reviewed By: walterddr Differential Revision: D25828491 Pulled By: seemethere fbshipit-source-id: 500ec96cb5f5da1321610002d5e3678f4b0b94b5
2021-01-07 22:11:01 +00:00
runtime-push: DOCKER_TAG := $(PYTORCH_VERSION)-runtime
docker: Refactor Dockerfile process for official images (#32515) Summary: ## Commit Message: Refactors Dockerfile to be as parallel as possible with caching and adds a new Makefile to build said Dockerfile. Also updated the README.md to reflect the changes as well as updated some of the verbage around running our latest Docker images. Adds the new Dockerfile process to our CircleCI workflows ## How to build: Building the new images is pretty simple, just requires `docker` > 18.06 since the new build process relies on `buildkit` caching and multi-stage build resolving. ### Development images For `runtime` images: ``` make -f docker.Makefile runtime-image ``` For `devel` images: ``` make -f docker.Makefile devel-image ``` Builds are tagged as follows: ```bash docker.io/${docker_user:-whoami}/pytorch:$(git describe --tags)-${image_type} ``` Example: ``` docker.io/seemethere/pytorch:v1.4.0a0-2225-g9eba97b61d-runtime ``` ### Official images Official images are the ones hosted on [`docker.io/pytorch/pytorch`](https://hub.docker.com/r/pytorch/pytorch) To do official images builds you can simply add set the `BUILD_TYPE` variable to `official` and it will do the correct build without building the local binaries: Example: ``` make -f docker.Makefile BUILD_TYPE=official runtime-image ``` ## How to push: Pushing is also super simple (And will automatically tag the right thing based off of the git tag): ``` make -f docker.Makefile runtime-push make -f docker.Makefile devel-push ``` Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: https://github.com/pytorch/pytorch/pull/32515 Differential Revision: D19558619 Pulled By: seemethere fbshipit-source-id: a06b25cd39ae9890751a60f8f36739ad6ab9ac99
2020-01-24 18:24:46 +00:00
runtime-push:
$(DOCKER_PUSH)
.PHONY: clean
clean:
-docker rmi -f $(shell docker images -q $(DOCKER_FULL_NAME))
Reference in a new issue Copy permalink