Update doc (add rl zoo)

docs/_static/img/colab-badge.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="117" height="20"><linearGradient id="b" x2="0" y2="100%"><stop offset="0" stop-color="#bbb" stop-opacity=".1"/><stop offset="1" stop-opacity=".1"/></linearGradient><clipPath id="a"><rect width="117" height="20" rx="3" fill="#fff"/></clipPath><g clip-path="url(#a)"><path fill="#555" d="M0 0h30v20H0z"/><path fill="#007ec6" d="M30 0h87v20H30z"/><path fill="url(#b)" d="M0 0h117v20H0z"/></g><g fill="#fff" text-anchor="middle" font-family="DejaVu Sans,Verdana,Geneva,sans-serif" font-size="110"><svg x="4px" y="0px" width="22px" height="20px" viewBox="-2 0 28 24" style="background-color: #fff;border-radius: 1px;"><path style="fill:#ef9008;" d="M1.977,16.77c-2.667-2.277-2.605-7.079,0-9.357C2.919,8.057,3.522,9.075,4.49,9.691c-1.152,1.6-1.146,3.201-0.004,4.803C3.522,15.111,2.918,16.126,1.977,16.77z"/><path style="fill:#fdba18;" d="M12.257,17.114c-1.767-1.633-2.485-3.658-2.118-6.02c0.451-2.91,2.139-4.893,4.946-5.678c2.565-0.718,4.964-0.217,6.878,1.819c-0.884,0.743-1.707,1.547-2.434,2.446C18.488,8.827,17.319,8.435,16,8.856c-2.404,0.767-3.046,3.241-1.494,5.644c-0.241,0.275-0.493,0.541-0.721,0.826C13.295,15.939,12.511,16.3,12.257,17.114z"/><path style="fill:#ef9008;" d="M19.529,9.682c0.727-0.899,1.55-1.703,2.434-2.446c2.703,2.783,2.701,7.031-0.005,9.764c-2.648,2.674-6.936,2.725-9.701,0.115c0.254-0.814,1.038-1.175,1.528-1.788c0.228-0.285,0.48-0.552,0.721-0.826c1.053,0.916,2.254,1.268,3.6,0.83C20.502,14.551,21.151,11.927,19.529,9.682z"/><path style="fill:#fdba18;" d="M4.49,9.691C3.522,9.075,2.919,8.057,1.977,7.413c2.209-2.398,5.721-2.942,8.476-1.355c0.555,0.32,0.719,0.606,0.285,1.128c-0.157,0.188-0.258,0.422-0.391,0.631c-0.299,0.47-0.509,1.067-0.929,1.371C8.933,9.539,8.523,8.847,8.021,8.746C6.673,8.475,5.509,8.787,4.49,9.691z"/><path style="fill:#fdba18;" d="M1.977,16.77c0.941-0.644,1.545-1.659,2.509-2.277c1.373,1.152,2.85,1.433,4.45,0.499c0.332-0.194,0.503-0.088,0.673,0.19c0.386,0.635,0.753,1.285,1.181,1.89c0.34,0.48,0.222,0.715-0.253,1.006C7.84,19.73,4.205,19.188,1.977,16.77z"/></svg><text x="245" y="140" transform="scale(.1)" textLength="30"> </text><text x="725" y="150" fill="#010101" fill-opacity=".3" transform="scale(.1)" textLength="770">Open in Colab</text><text x="725" y="140" transform="scale(.1)" textLength="770">Open in Colab</text></g> </svg>

docs/guide/examples.rst

@@ -12,6 +12,7 @@ notebooks:
 -  `Full Tutorial <https://github.com/araffin/rl-tutorial-jnrr19>`_
 -  `All Notebooks <https://github.com/Stable-Baselines-Team/rl-colab-notebooks/tree/sb3>`_
 -  `Getting Started`_
+-  `RL Baselines zoo`_
 
 
 .. -  `Training, Saving, Loading`_
@@ -20,16 +21,15 @@ notebooks:
 .. -  `Atari Games`_
 .. -  `Breakout`_ (trained agent included)
 .. -  `Hindsight Experience Replay`_
-.. -  `RL Baselines zoo`_
 
 .. _Getting Started: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/stable_baselines_getting_started.ipynb
-.. _Training, Saving, Loading: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/saving_loading_dqn.ipynb
-.. _Multiprocessing: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/multiprocessing_rl.ipynb
-.. _Monitor Training and Plotting: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/monitor_training.ipynb
-.. _Atari Games: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/atari_games.ipynb
-.. _Breakout: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/breakout.ipynb
-.. _Hindsight Experience Replay: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/stable_baselines_her.ipynb
-.. _RL Baselines zoo: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/rl-baselines-zoo.ipynb
+.. _Training, Saving, Loading: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/saving_loading_dqn.ipynb
+.. _Multiprocessing: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/multiprocessing_rl.ipynb
+.. _Monitor Training and Plotting: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/monitor_training.ipynb
+.. _Atari Games: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/atari_games.ipynb
+.. _Breakout: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/breakout.ipynb
+.. _Hindsight Experience Replay: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/stable_baselines_her.ipynb
+.. _RL Baselines zoo: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/rl-baselines-zoo.ipynb
 
 .. |colab| image:: ../_static/img/colab.svg
 
@@ -38,9 +38,8 @@ Basic Usage: Training, Saving, Loading
 
 In the following example, we will train, save and load a DQN model on the Lunar Lander environment.
 
-.. .. image:: ../_static/img/try_it.png
-..    :scale: 30 %
-..    :target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/saving_loading_dqn.ipynb
+.. .. image:: ../_static/img/colab-badge.svg
+..    :target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/saving_loading_dqn.ipynb
 
 
 .. figure:: https://cdn-images-1.medium.com/max/960/1*f4VZPKOI0PYNWiwt0la0Rg.gif
@@ -93,9 +92,8 @@ In the following example, we will train, save and load a DQN model on the Lunar
 Multiprocessing: Unleashing the Power of Vectorized Environments
 ----------------------------------------------------------------
 ..
-.. .. image:: ../_static/img/try_it.png
-..    :scale: 30 %
-..    :target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/multiprocessing_rl.ipynb
+.. .. image:: ../_static/img/colab-badge.svg
+..    :target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/multiprocessing_rl.ipynb
 
 .. figure:: https://cdn-images-1.medium.com/max/960/1*h4WTQNVIsvMXJTCpXm_TAw.gif
 
@@ -162,9 +160,8 @@ This could be useful when you want to monitor training, for instance display liv
 learning curves in Tensorboard (or in Visdom) or save the best agent.
 If your callback returns False, training is aborted early.
 
-.. .. image:: ../_static/img/try_it.png
-..    :scale: 30 %
-..    :target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/monitor_training.ipynb
+.. .. image:: ../_static/img/colab-badge.svg
+..    :target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/monitor_training.ipynb
 ..
 .. .. figure:: ../_static/img/learning_curve.png
 ..
@@ -270,9 +267,8 @@ Training a RL agent on Atari games is straightforward thanks to ``make_atari_env
 It will do `all the preprocessing <https://danieltakeshi.github.io/2016/11/25/frame-skipping-and-preprocessing-for-deep-q-networks-on-atari-2600-games/>`_
 and multiprocessing for you.
 
-.. .. image:: ../_static/img/try_it.png
-..    :scale: 30 %
-..    :target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/master/atari_games.ipynb
+.. .. image:: ../_static/img/colab-badge.svg
+..    :target: https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/atari_games.ipynb
 ..
 
 .. code-block:: python

docs/guide/rl_zoo.rst

@@ -0,0 +1,108 @@
+.. _rl_zoo:
+
+==================
+RL Baselines3 Zoo
+==================
+
+`RL Baselines3 Zoo <https://github.com/DLR-RM/rl-baselines3-zoo>`_. is a collection of pre-trained Reinforcement Learning agents using
+Stable-Baselines3.
+It also provides basic scripts for training, evaluating agents, tuning hyperparameters and recording videos.
+
+Goals of this repository:
+
+1. Provide a simple interface to train and enjoy RL agents
+2. Benchmark the different Reinforcement Learning algorithms
+3. Provide tuned hyperparameters for each environment and RL algorithm
+4. Have fun with the trained agents!
+
+Installation
+------------
+
+1. Clone the repository:
+
+::
+
+  git clone --recursive https://github.com/DLR-RM/rl-baselines3-zoo
+  cd rl-baselines3-zoo/
+
+
+.. note::
+
+	You can remove the ``--recursive`` option if you don't want to download the trained agents
+
+
+2. Install dependencies
+::
+
+   apt-get install swig cmake ffmpeg
+   pip install -r requirements.txt
+
+
+Train an Agent
+--------------
+
+The hyperparameters for each environment are defined in
+``hyperparameters/algo_name.yml``.
+
+If the environment exists in this file, then you can train an agent
+using:
+
+::
+
+ python train.py --algo algo_name --env env_id
+
+For example (with evaluation and checkpoints):
+
+::
+
+ python train.py --algo ppo2 --env CartPole-v1 --eval-freq 10000 --save-freq 50000
+
+
+Continue training (here, load pretrained agent for Breakout and continue
+training for 5000 steps):
+
+::
+
+ python train.py --algo a2c --env BreakoutNoFrameskip-v4 -i trained_agents/a2c/BreakoutNoFrameskip-v4_1/BreakoutNoFrameskip-v4.zip -n 5000
+
+
+Enjoy a Trained Agent
+---------------------
+
+If the trained agent exists, then you can see it in action using:
+
+::
+
+  python enjoy.py --algo algo_name --env env_id
+
+For example, enjoy A2C on Breakout during 5000 timesteps:
+
+::
+
+  python enjoy.py --algo a2c --env BreakoutNoFrameskip-v4 --folder rl-trained_agents/ -n 5000
+
+
+Hyperparameter Optimization
+---------------------------
+
+We use `Optuna <https://optuna.org/>`_ for optimizing the hyperparameters.
+
+
+Tune the hyperparameters for PPO, using a random sampler and median pruner, 2 parallels jobs,
+with a budget of 1000 trials and a maximum of 50000 steps:
+
+::
+
+  python train.py --algo ppo --env MountainCar-v0 -n 50000 -optimize --n-trials 1000 --n-jobs 2 \
+    --sampler random --pruner median
+
+
+Colab Notebook: Try it Online!
+------------------------------
+
+You can train agents online using Google `colab notebook <https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/rl-baselines-zoo.ipynb>`_.
+
+
+.. note::
+
+	You can find more information about the rl baselines3 zoo in the repo `README <https://github.com/DLR-RM/rl-baselines3-zoo>`_. For instance, how to record a video of a trained agent.

docs/index.rst

@@ -42,6 +42,7 @@ Main Features
    guide/custom_env
    guide/custom_policy
    guide/callbacks
+   guide/rl_zoo
    guide/migration
    guide/checking_nan