diff --git a/README.md b/README.md index f727547..fd6c8db 100644 --- a/README.md +++ b/README.md @@ -77,7 +77,7 @@ Documentation: https://stable-baselines3.readthedocs.io/en/master/guide/rl_zoo.h We implement experimental features in a separate contrib repository: [SB3-Contrib](https://github.com/Stable-Baselines-Team/stable-baselines3-contrib) -This allows SB3 to maintain a stable and compact core, while still providing the latest features, like Truncated Quantile Critics (TQC), Quantile Regression DQN (QR-DQN) or PPO with invalid action masking (Maskable PPO). +This allows SB3 to maintain a stable and compact core, while still providing the latest features, like Recurrent PPO (PPO LSTM), Truncated Quantile Critics (TQC), Quantile Regression DQN (QR-DQN) or PPO with invalid action masking (Maskable PPO). Documentation is available online: [https://sb3-contrib.readthedocs.io/](https://sb3-contrib.readthedocs.io/) @@ -122,7 +122,7 @@ from stable_baselines3 import PPO env = gym.make("CartPole-v1") model = PPO("MlpPolicy", env, verbose=1) -model.learn(total_timesteps=10000) +model.learn(total_timesteps=10_000) obs = env.reset() for i in range(1000): @@ -140,7 +140,7 @@ Or just train a model with a one liner if [the environment is registered in Gym] ```python from stable_baselines3 import PPO -model = PPO('MlpPolicy', 'CartPole-v1').learn(10000) +model = PPO("MlpPolicy", "CartPole-v1").learn(10_000) ``` Please read the [documentation](https://stable-baselines3.readthedocs.io/) for more examples. @@ -172,6 +172,7 @@ All the following examples can be executed online using Google colab notebooks: | HER | :x: | :heavy_check_mark: | :heavy_check_mark: | :x: | :x: | :x: | | PPO | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | QR-DQN[1](#f1) | :x: | :x: | :heavy_check_mark: | :x: | :x: | :heavy_check_mark: | +| RecurrentPPO[1](#f1) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | | SAC | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: | | TD3 | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: | | TQC[1](#f1) | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: | diff --git a/docs/guide/algos.rst b/docs/guide/algos.rst index 5d1bf08..55aba35 100644 --- a/docs/guide/algos.rst +++ b/docs/guide/algos.rst @@ -15,6 +15,7 @@ DQN ❌ ✔️ ❌ ❌ HER ✔️ ✔️ ❌ ❌ ❌ PPO ✔️ ✔️ ✔️ ✔️ ✔️ QR-DQN [#f1]_ ❌ ️ ✔️ ❌ ❌ ✔️ +RecurrentPPO [#f1]_ ✔️ ✔️ ✔️ ✔️ ✔️ SAC ✔️ ❌ ❌ ❌ ✔️ TD3 ✔️ ❌ ❌ ❌ ✔️ TQC [#f1]_ ✔️ ❌ ❌ ❌ ✔️ diff --git a/docs/guide/sb3_contrib.rst b/docs/guide/sb3_contrib.rst index 1dfa912..445832c 100644 --- a/docs/guide/sb3_contrib.rst +++ b/docs/guide/sb3_contrib.rst @@ -8,7 +8,7 @@ We implement experimental features in a separate contrib repository: `SB3-Contrib`_ This allows Stable-Baselines3 (SB3) to maintain a stable and compact core, while still -providing the latest features, like Truncated Quantile Critics (TQC), Augmented Random Search (ARS), Trust Region Policy Optimization (TRPO) or +providing the latest features, like RecurrentPPO (PPO LSTM), Truncated Quantile Critics (TQC), Augmented Random Search (ARS), Trust Region Policy Optimization (TRPO) or Quantile Regression DQN (QR-DQN). Why create this repository? @@ -38,9 +38,11 @@ See documentation for the full list of included features. - `Augmented Random Search (ARS) `_ - `Quantile Regression DQN (QR-DQN)`_ +- `PPO with invalid action masking (Maskable PPO) `_ +- `PPO with recurrent policy (RecurrentPPO aka PPO LSTM) `_ - `Truncated Quantile Critics (TQC)`_ - `Trust Region Policy Optimization (TRPO) `_ -- `PPO with invalid action masking (Maskable PPO) `_ + **Gym Wrappers**: diff --git a/docs/misc/changelog.rst b/docs/misc/changelog.rst index abd7c0c..1c88691 100644 --- a/docs/misc/changelog.rst +++ b/docs/misc/changelog.rst @@ -4,7 +4,7 @@ Changelog ========== -Release 1.5.1a7 (WIP) +Release 1.5.1a8 (WIP) --------------------------- Breaking Changes: @@ -18,6 +18,8 @@ New Features: SB3-Contrib ^^^^^^^^^^^ +- Added Recurrent PPO (PPO LSTM). See https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/pull/53 + Bug Fixes: ^^^^^^^^^^ diff --git a/docs/modules/ppo.rst b/docs/modules/ppo.rst index d32986c..a562aa2 100644 --- a/docs/modules/ppo.rst +++ b/docs/modules/ppo.rst @@ -15,7 +15,7 @@ For that, ppo uses clipping to avoid too large update. .. note:: PPO contains several modifications from the original algorithm not documented - by OpenAI: advantages are normalized and value function can be also clipped . + by OpenAI: advantages are normalized and value function can be also clipped. Notes @@ -31,6 +31,16 @@ Notes Can I use? ---------- +.. note:: + + A recurrent version of PPO is available in our contrib repo: https://sb3-contrib.readthedocs.io/en/master/modules/ppo_recurrent.html + + However we advise users to start with simple frame-stacking as a simpler, faster + and usually competitive alternative, more info in our report: https://wandb.ai/sb3/no-vel-envs/reports/PPO-vs-RecurrentPPO-aka-PPO-LSTM-on-environments-with-masked-velocity--VmlldzoxOTI4NjE4 + See also `Procgen paper appendix Fig 11. `_. + In practice, you can stack multiple observations using ``VecFrameStack``. + + - Recurrent policies: ❌ - Multi processing: ✔️ - Gym spaces: diff --git a/stable_baselines3/version.txt b/stable_baselines3/version.txt index e39732b..511e75b 100644 --- a/stable_baselines3/version.txt +++ b/stable_baselines3/version.txt @@ -1 +1 @@ -1.5.1a7 +1.5.1a8