Update doc: SB3 Contrib RecurrentPPO (#927)

* Update doc: contrib update

* Update docs/misc/changelog.rst

Co-authored-by: Anssi <kaneran21@hotmail.com>

* Address Anssi comments

Co-authored-by: Anssi <kaneran21@hotmail.com>

README.md

@@ -77,7 +77,7 @@ Documentation: https://stable-baselines3.readthedocs.io/en/master/guide/rl_zoo.h
 
 We implement experimental features in a separate contrib repository: [SB3-Contrib](https://github.com/Stable-Baselines-Team/stable-baselines3-contrib)
 
-This allows SB3 to maintain a stable and compact core, while still providing the latest features, like Truncated Quantile Critics (TQC), Quantile Regression DQN (QR-DQN) or PPO with invalid action masking (Maskable PPO).
+This allows SB3 to maintain a stable and compact core, while still providing the latest features, like Recurrent PPO (PPO LSTM), Truncated Quantile Critics (TQC), Quantile Regression DQN (QR-DQN) or PPO with invalid action masking (Maskable PPO).
 
 Documentation is available online: [https://sb3-contrib.readthedocs.io/](https://sb3-contrib.readthedocs.io/)
 
@@ -122,7 +122,7 @@ from stable_baselines3 import PPO
 env = gym.make("CartPole-v1")
 
 model = PPO("MlpPolicy", env, verbose=1)
-model.learn(total_timesteps=10000)
+model.learn(total_timesteps=10_000)
 
 obs = env.reset()
 for i in range(1000):
@@ -140,7 +140,7 @@ Or just train a model with a one liner if [the environment is registered in Gym]
 ```python
 from stable_baselines3 import PPO
 
-model = PPO('MlpPolicy', 'CartPole-v1').learn(10000)
+model = PPO("MlpPolicy", "CartPole-v1").learn(10_000)
 ```
 
 Please read the [documentation](https://stable-baselines3.readthedocs.io/) for more examples.
@@ -172,6 +172,7 @@ All the following examples can be executed online using Google colab notebooks:
 | HER   | :x: | :heavy_check_mark: | :heavy_check_mark: | :x:                 | :x:                | :x: |
 | PPO   | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:  | :heavy_check_mark: | :heavy_check_mark: |
 | QR-DQN<sup>[1](#f1)</sup>  | :x: | :x: | :heavy_check_mark: | :x:                 | :x:                | :heavy_check_mark: |
+| RecurrentPPO<sup>[1](#f1)</sup>   | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:  | :heavy_check_mark: | :heavy_check_mark: |
 | SAC   | :x: | :heavy_check_mark: | :x:                | :x:                 | :x:                | :heavy_check_mark: |
 | TD3   | :x: | :heavy_check_mark: | :x:                | :x:                 | :x:                | :heavy_check_mark: |
 | TQC<sup>[1](#f1)</sup>   | :x: | :heavy_check_mark: | :x:                | :x:                 | :x: | :heavy_check_mark: |

docs/guide/algos.rst

@@ -15,6 +15,7 @@ DQN                  ❌           ✔️            ❌                ❌
 HER                  ✔️           ✔️            ❌                ❌               ❌
 PPO                  ✔️           ✔️            ✔️                 ✔️               ✔️
 QR-DQN [#f1]_        ❌          ️ ✔️            ❌                ❌               ✔️
+RecurrentPPO [#f1]_  ✔️           ✔️             ✔️                ✔️               ✔️
 SAC                  ✔️           ❌            ❌                ❌               ✔️
 TD3                  ✔️           ❌            ❌                ❌               ✔️
 TQC [#f1]_           ✔️           ❌            ❌                ❌               ✔️

docs/guide/sb3_contrib.rst

@@ -8,7 +8,7 @@ We implement experimental features in a separate contrib repository:
 `SB3-Contrib`_
 
 This allows Stable-Baselines3 (SB3) to maintain a stable and compact core, while still
-providing the latest features, like Truncated Quantile Critics (TQC), Augmented Random Search (ARS), Trust Region Policy Optimization (TRPO) or
+providing the latest features, like RecurrentPPO (PPO LSTM), Truncated Quantile Critics (TQC), Augmented Random Search (ARS), Trust Region Policy Optimization (TRPO) or
 Quantile Regression DQN (QR-DQN).
 
 Why create this repository?
@@ -38,9 +38,11 @@ See documentation for the full list of included features.
 
 - `Augmented Random Search (ARS) <https://arxiv.org/abs/1803.07055>`_
 - `Quantile Regression DQN (QR-DQN)`_
+- `PPO with invalid action masking (Maskable PPO) <https://arxiv.org/abs/2006.14171>`_
+- `PPO with recurrent policy (RecurrentPPO aka PPO LSTM) <https://ppo-details.cleanrl.dev//2021/11/05/ppo-implementation-details/>`_
 - `Truncated Quantile Critics (TQC)`_
 - `Trust Region Policy Optimization (TRPO) <https://arxiv.org/abs/1502.05477>`_
-- `PPO with invalid action masking (Maskable PPO) <https://arxiv.org/abs/2006.14171>`_
+
 
 **Gym Wrappers**:
 

docs/misc/changelog.rst

@@ -4,7 +4,7 @@ Changelog
 ==========
 
 
-Release 1.5.1a7 (WIP)
+Release 1.5.1a8 (WIP)
 ---------------------------
 
 Breaking Changes:
@@ -18,6 +18,8 @@ New Features:
 
 SB3-Contrib
 ^^^^^^^^^^^
+- Added Recurrent PPO (PPO LSTM). See https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/pull/53
+
 
 Bug Fixes:
 ^^^^^^^^^^

docs/modules/ppo.rst

@@ -15,7 +15,7 @@ For that, ppo uses clipping to avoid too large update.
 .. note::
 
   PPO contains several modifications from the original algorithm not documented
-  by OpenAI: advantages are normalized and value function can be also clipped .
+  by OpenAI: advantages are normalized and value function can be also clipped.
 
 
 Notes
@@ -31,6 +31,16 @@ Notes
 Can I use?
 ----------
 
+.. note::
+
+  A recurrent version of PPO is available in our contrib repo: https://sb3-contrib.readthedocs.io/en/master/modules/ppo_recurrent.html
+
+  However we advise users to start with simple frame-stacking as a simpler, faster
+  and usually competitive alternative, more info in our report: https://wandb.ai/sb3/no-vel-envs/reports/PPO-vs-RecurrentPPO-aka-PPO-LSTM-on-environments-with-masked-velocity--VmlldzoxOTI4NjE4
+  See also `Procgen paper appendix Fig 11. <https://arxiv.org/abs/1912.01588>`_.
+  In practice, you can stack multiple observations using ``VecFrameStack``.
+
+
 -  Recurrent policies: ❌
 -  Multi processing: ✔️
 -  Gym spaces:

stable_baselines3/version.txt

@@ -1 +1 @@
-1.5.1a7
+1.5.1a8