Add dockerfile + update doc

.dockerignore

@@ -0,0 +1 @@
+.gitignore

.github/ISSUE_TEMPLATE/issue-template.md

@@ -5,6 +5,7 @@ about: How to create an issue for this repository
 ---
 
 **Important Note: We do not do technical support, nor consulting** and don't answer personal questions per email.
+Please post your question on [reddit](https://www.reddit.com/r/reinforcementlearning/) or [stack overflow](https://stackoverflow.com/) in that case.
 
 If you have any questions, feel free to create an issue with the tag [question].  
 If you wish to suggest an enhancement or feature request, add the tag [feature request].  

CONTRIBUTING.md

@@ -0,0 +1,121 @@
+## Contributing to Stable-Baselines3
+
+If you are interested in contributing to Stable-Baselines, your contributions will fall
+into two categories:
+1. You want to propose a new Feature and implement it
+    - Create an issue about your intended feature, and we shall discuss the design and
+    implementation. Once we agree that the plan looks good, go ahead and implement it.
+2. You want to implement a feature or bug-fix for an outstanding issue
+    - Look at the outstanding issues here: https://github.com/DLR-RM/stable-baselines3/issues
+    - Pick an issue or feature and comment on the task that you want to work on this feature.
+    - If you need more context on a particular issue, please ask and we shall provide.
+
+Once you finish implementing a feature or bug-fix, please send a Pull Request to
+https://github.com/DLR-RM/stable-baselines3
+
+
+If you are not familiar with creating a Pull Request, here are some guides:
+- http://stackoverflow.com/questions/14680711/how-to-do-a-github-pull-request
+- https://help.github.com/articles/creating-a-pull-request/
+
+
+## Developing Stable-Baselines3
+
+To develop Stable-Baselines3 on your machine, here are some tips:
+
+1. Clone a copy of Stable-Baselines3 from source:
+
+```bash
+git clone https://github.com/DLR-RM/stable-baselines3
+cd stable-baselines3/
+```
+
+2. Install Stable-Baselines3 in develop mode, with support for building the docs and running tests:
+
+```bash
+pip install -e .[docs,tests,extra]
+```
+
+## Codestyle
+
+We follow the [PEP8 codestyle](https://www.python.org/dev/peps/pep-0008/). Please order the imports as follows:
+
+1. built-in
+2. packages
+3. current module
+
+with one space between each,  that gives for instance:
+```python
+import os
+import warnings
+
+import numpy as np
+
+from stable_baselines3 import PPO
+```
+
+In general, we recommend using pycharm to format everything in an efficient way.
+
+Please document each function/method and [type](https://google.github.io/pytype/user_guide.html) them using the following template:
+
+```python
+
+def my_function(arg1: type1, arg2: type2) -> returntype:
+    """
+    Short description of the function.
+
+    :param arg1: (type1) describe what is arg1
+    :param arg2: (type2) describe what is arg2
+    :return: (returntype) describe what is returned
+    """
+    ...
+    return my_variable
+```
+
+## Pull Request (PR)
+
+Before proposing a PR, please open an issue, where the feature will be discussed. This prevent from duplicated PR to be proposed and also ease the code review process.
+
+Each PR need to be reviewed and accepted by at least one of the maintainers (@hill-a, @araffin, @erniejunior, @AdamGleave or @Miffyli).
+A PR must pass the Continuous Integration tests (travis + codacy) to be merged with the master branch.
+
+Note: in rare cases, we can create exception for codacy failure.
+
+## Test
+
+All new features must add tests in the `tests/` folder ensuring that everything works fine.
+We use [pytest](https://pytest.org/).
+Also, when a bug fix is proposed, tests should be added to avoid regression.
+
+To run tests with `pytest`:
+
+```
+make pytest
+```
+
+Type checking with `pytype`:
+
+```
+make type
+```
+
+Build the documentation:
+
+```
+make doc
+```
+
+Check documentation spelling (you need to install `sphinxcontrib.spelling` package for that):
+
+```
+make spelling
+```
+
+
+## Changelog and Documentation
+
+Please do not forget to update the changelog (`docs/misc/changelog.rst`) and add documentation if needed.
+A README is present in the `docs/` folder for instructions on how to build the documentation.
+
+
+Credits: this contributing guide is based on the [PyTorch](https://github.com/pytorch/pytorch/) one.

Dockerfile

@@ -0,0 +1,48 @@
+ARG PARENT_IMAGE
+FROM $PARENT_IMAGE
+ARG PYTORCH_DEPS=cpuonly
+ARG PYTHON_VERSION=3.6
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+         build-essential \
+         cmake \
+         git \
+         curl \
+         ca-certificates \
+         libjpeg-dev \
+         libpng-dev && \
+     rm -rf /var/lib/apt/lists/*
+
+# Install anaconda abd dependencies
+RUN curl -o ~/miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
+     chmod +x ~/miniconda.sh && \
+     ~/miniconda.sh -b -p /opt/conda && \
+     rm ~/miniconda.sh && \
+     /opt/conda/bin/conda install -y python=$PYTHON_VERSION numpy pyyaml scipy ipython mkl mkl-include && \
+     /opt/conda/bin/conda install -y pytorch $PYTORCH_DEPS -c pytorch && \
+     /opt/conda/bin/conda clean -ya
+ENV PATH /opt/conda/bin:$PATH
+
+ENV CODE_DIR /root/code
+
+# Copy setup file only to install dependencies
+COPY ./setup.py ${CODE_DIR}/stable-baselines3/setup.py
+COPY ./stable_baselines3/version.txt ${CODE_DIR}/stable-baselines3/stable_baselines3/version.txt
+
+RUN \
+    cd ${CODE_DIR}/stable-baselines3 3&& \
+    pip install -e .[extra,tests,docs] && \
+    rm -rf $HOME/.cache/pip
+
+
+# Codacy deps
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    default-jre \
+    jq && \
+    rm -rf /var/lib/apt/lists/*
+
+# Codacy code coverage report: used for partial code coverage reporting
+RUN cd $CODE_DIR && \
+    curl -Ls -o codacy-coverage-reporter.jar "$(curl -Ls https://api.github.com/repos/codacy/codacy-coverage-reporter/releases/latest | jq -r '.assets | map({name, browser_download_url} | select(.name | (startswith("codacy-coverage-reporter") and contains("assembly") and endswith(".jar")))) | .[0].browser_download_url')"
+
+CMD /bin/bash

Makefile

@@ -17,16 +17,15 @@ clean:
 
 .PHONY: clean spelling doc
 
-# TODO: create Dockerfile
+# Build docker images
-# # Build docker images
+# If you do export RELEASE=True, it will also push them
-# # If you do export RELEASE=True, it will also push them
+docker: docker-cpu docker-gpu
-# docker: docker-cpu docker-gpu
+
-#
+docker-cpu:
-# docker-cpu:
+	./scripts/build_docker.sh
-# 	./scripts/build_docker.sh
+
-#
+docker-gpu:
-# docker-gpu:
+	USE_GPU=True ./scripts/build_docker.sh
-# 	USE_GPU=True ./scripts/build_docker.sh
 
 # PyPi package release
 release:

docs/guide/install.rst

@@ -56,97 +56,97 @@ To contribute to Stable-Baselines3, with support for running tests and building
     pip install -e .[docs,tests,extra]
 
 
-.. Using Docker Images
+Using Docker Images
-.. -------------------
+-------------------
-..
+
-.. If you are looking for docker images with stable-baselines already installed in it,
+If you are looking for docker images with stable-baselines already installed in it,
-.. we recommend using images from `RL Baselines3 Zoo <https://github.com/DLR-RM/rl-baselines3-zoo>`_.
+we recommend using images from `RL Baselines3 Zoo <https://github.com/DLR-RM/rl-baselines3-zoo>`_.
-..
+
-.. Otherwise, the following images contained all the dependencies for stable-baselines3 but not the stable-baselines3 package itself.
+Otherwise, the following images contained all the dependencies for stable-baselines3 but not the stable-baselines3 package itself.
-.. They are made for development.
+They are made for development.
-..
+
-.. Use Built Images
+Use Built Images
-.. ~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~
-..
+
-.. GPU image (requires `nvidia-docker`_):
+GPU image (requires `nvidia-docker`_):
-..
+
-.. .. code-block:: bash
+.. code-block:: bash
-..
+
-..    docker pull stablebaselines/stable-baselines3
+   docker pull stablebaselines/stable-baselines3
-..
+
-.. CPU only:
+CPU only:
-..
+
-.. .. code-block:: bash
+.. code-block:: bash
-..
+
-..    docker pull stablebaselines/stable-baselines3-cpu
+   docker pull stablebaselines/stable-baselines3-cpu
-..
+
-.. Build the Docker Images
+Build the Docker Images
-.. ~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~
-..
+
-.. Build GPU image (with nvidia-docker):
+Build GPU image (with nvidia-docker):
-..
+
-.. .. code-block:: bash
+.. code-block:: bash
-..
+
-..    make docker-gpu
+   make docker-gpu
-..
+
-.. Build CPU image:
+Build CPU image:
-..
+
-.. .. code-block:: bash
+.. code-block:: bash
-..
+
-..    make docker-cpu
+   make docker-cpu
-..
+
-.. Note: if you are using a proxy, you need to pass extra params during
+Note: if you are using a proxy, you need to pass extra params during
-.. build and do some `tweaks`_:
+build and do some `tweaks`_:
-..
+
-.. .. code-block:: bash
+.. code-block:: bash
-..
+
-..    --network=host --build-arg HTTP_PROXY=http://your.proxy.fr:8080/ --build-arg http_proxy=http://your.proxy.fr:8080/ --build-arg HTTPS_PROXY=https://your.proxy.fr:8080/ --build-arg https_proxy=https://your.proxy.fr:8080/
+   --network=host --build-arg HTTP_PROXY=http://your.proxy.fr:8080/ --build-arg http_proxy=http://your.proxy.fr:8080/ --build-arg HTTPS_PROXY=https://your.proxy.fr:8080/ --build-arg https_proxy=https://your.proxy.fr:8080/
-..
+
-.. Run the images (CPU/GPU)
+Run the images (CPU/GPU)
-.. ~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~
-..
+
-.. Run the nvidia-docker GPU image
+Run the nvidia-docker GPU image
-..
+
-.. .. code-block:: bash
+.. code-block:: bash
-..
+
-..    docker run -it --runtime=nvidia --rm --network host --ipc=host --name test --mount src="$(pwd)",target=/root/code/stable-baselines,type=bind stablebaselines/stable-baselines bash -c 'cd /root/code/stable-baselines/ && pytest tests/'
+   docker run -it --runtime=nvidia --rm --network host --ipc=host --name test --mount src="$(pwd)",target=/root/code/stable-baselines3,type=bind stablebaselines/stable-baselines3 bash -c 'cd /root/code/stable-baselines3/ && pytest tests/'
-..
+
-.. Or, with the shell file:
+Or, with the shell file:
-..
+
-.. .. code-block:: bash
+.. code-block:: bash
-..
+
-..    ./scripts/run_docker_gpu.sh pytest tests/
+   ./scripts/run_docker_gpu.sh pytest tests/
-..
+
-.. Run the docker CPU image
+Run the docker CPU image
-..
+
-.. .. code-block:: bash
+.. code-block:: bash
-..
+
-..    docker run -it --rm --network host --ipc=host --name test --mount src="$(pwd)",target=/root/code/stable-baselines,type=bind stablebaselines/stable-baselines-cpu bash -c 'cd /root/code/stable-baselines/ && pytest tests/'
+   docker run -it --rm --network host --ipc=host --name test --mount src="$(pwd)",target=/root/code/stable-baselines3,type=bind stablebaselines/stable-baselines3-cpu bash -c 'cd /root/code/stable-baselines3/ && pytest tests/'
-..
+
-.. Or, with the shell file:
+Or, with the shell file:
-..
+
-.. .. code-block:: bash
+.. code-block:: bash
-..
+
-..    ./scripts/run_docker_cpu.sh pytest tests/
+   ./scripts/run_docker_cpu.sh pytest tests/
-..
+
-.. Explanation of the docker command:
+Explanation of the docker command:
-..
+
-.. -  ``docker run -it`` create an instance of an image (=container), and
+-  ``docker run -it`` create an instance of an image (=container), and
-..    run it interactively (so ctrl+c will work)
+   run it interactively (so ctrl+c will work)
-.. -  ``--rm`` option means to remove the container once it exits/stops
+-  ``--rm`` option means to remove the container once it exits/stops
-..    (otherwise, you will have to use ``docker rm``)
+   (otherwise, you will have to use ``docker rm``)
-.. -  ``--network host`` don't use network isolation, this allow to use
+-  ``--network host`` don't use network isolation, this allow to use
-..    tensorboard/visdom on host machine
+   tensorboard/visdom on host machine
-.. -  ``--ipc=host`` Use the host system’s IPC namespace. IPC (POSIX/SysV IPC) namespace provides
+-  ``--ipc=host`` Use the host system’s IPC namespace. IPC (POSIX/SysV IPC) namespace provides
-..    separation of named shared memory segments, semaphores and message
+   separation of named shared memory segments, semaphores and message
-..    queues.
+   queues.
-.. -  ``--name test`` give explicitly the name ``test`` to the container,
+-  ``--name test`` give explicitly the name ``test`` to the container,
-..    otherwise it will be assigned a random name
+   otherwise it will be assigned a random name
-.. -  ``--mount src=...`` give access of the local directory (``pwd``
+-  ``--mount src=...`` give access of the local directory (``pwd``
-..    command) to the container (it will be map to ``/root/code/stable-baselines``), so
+   command) to the container (it will be map to ``/root/code/stable-baselines``), so
-..    all the logs created in the container in this folder will be kept
+   all the logs created in the container in this folder will be kept
-.. -  ``bash -c '...'`` Run command inside the docker image, here run the tests
+-  ``bash -c '...'`` Run command inside the docker image, here run the tests
-..    (``pytest tests/``)
+   (``pytest tests/``)
-..
+
-.. .. _nvidia-docker: https://github.com/NVIDIA/nvidia-docker
+.. _nvidia-docker: https://github.com/NVIDIA/nvidia-docker
-.. .. _tweaks: https://stackoverflow.com/questions/23111631/cannot-download-docker-images-behind-a-proxy
+.. _tweaks: https://stackoverflow.com/questions/23111631/cannot-download-docker-images-behind-a-proxy

scripts/build_docker.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+CPU_PARENT=ubuntu:16.04
+GPU_PARENT=nvidia/cuda:10.1-cudnn7-runtime-ubuntu16.04
+
+TAG=stablebaselines/stable-baselines3
+VERSION=$(cat ./stable_baselines3/version.txt)
+
+if [[ ${USE_GPU} == "True" ]]; then
+  PARENT=${GPU_PARENT}
+  PYTORCH_DEPS="cudatoolkit=10.1"
+else
+  PARENT=${CPU_PARENT}
+  PYTORCH_DEPS="cpuonly"
+  TAG="${TAG}-cpu"
+fi
+
+echo "docker build --build-arg PARENT_IMAGE=${PARENT} --build-arg PYTORCH_DEPS=${PYTORCH_DEPS} -t ${TAG}:${VERSION} ."
+docker build --build-arg PARENT_IMAGE=${PARENT} --build-arg PYTORCH_DEPS=${PYTORCH_DEPS} -t ${TAG}:${VERSION} .
+docker tag ${TAG}:${VERSION} ${TAG}:latest
+
+if [[ ${RELEASE} == "True" ]]; then
+  docker push ${TAG}:${VERSION}
+  docker push ${TAG}:latest
+fi

scripts/run_docker_cpu.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+# Launch an experiment using the docker cpu image
+
+cmd_line="$@"
+
+echo "Executing in the docker (cpu image):"
+echo $cmd_line
+
+docker run -it --rm --network host --ipc=host \
+ --mount src=$(pwd),target=/root/code/stable-baselines3,type=bind stablebaselines/stable-baselines3-cpu:latest \
+  bash -c "cd /root/code/stable-baselines3/ && $cmd_line"

scripts/run_docker_gpu.sh

@@ -0,0 +1,19 @@
+#!/bin/bash
+# Launch an experiment using the docker gpu image
+
+cmd_line="$@"
+
+echo "Executing in the docker (gpu image):"
+echo $cmd_line
+
+# TODO: always use new-style once sufficiently widely used (probably 2021 onwards)
+if [ -x "$(which nvidia-docker)" ]; then
+  # old-style nvidia-docker2
+  NVIDIA_ARG="--runtime=nvidia"
+else
+  NVIDIA_ARG="--gpus all"
+fi
+
+docker run -it ${NVIDIA_ARG} --rm --network host --ipc=host \
+  --mount src=$(pwd),target=/root/code/stable-baselines3,type=bind stablebaselines/stable-baselines3:latest \
+  bash -c "cd /root/code/stable-baselines3/ && $cmd_line"