mirror of
https://github.com/saymrwulf/stable-baselines3.git
synced 2026-11-18 09:54:10 +00:00
Update doc: SB3 Contrib RecurrentPPO (#927)
* Update doc: contrib update * Update docs/misc/changelog.rst Co-authored-by: Anssi <kaneran21@hotmail.com> * Address Anssi comments Co-authored-by: Anssi <kaneran21@hotmail.com>
This commit is contained in:
parent
4b89fbf283
commit
d68f0a2411
6 changed files with 24 additions and 8 deletions
|
|
@ -77,7 +77,7 @@ Documentation: https://stable-baselines3.readthedocs.io/en/master/guide/rl_zoo.h
|
||||||
|
|
||||||
We implement experimental features in a separate contrib repository: [SB3-Contrib](https://github.com/Stable-Baselines-Team/stable-baselines3-contrib)
|
We implement experimental features in a separate contrib repository: [SB3-Contrib](https://github.com/Stable-Baselines-Team/stable-baselines3-contrib)
|
||||||
|
|
||||||
This allows SB3 to maintain a stable and compact core, while still providing the latest features, like Truncated Quantile Critics (TQC), Quantile Regression DQN (QR-DQN) or PPO with invalid action masking (Maskable PPO).
|
This allows SB3 to maintain a stable and compact core, while still providing the latest features, like Recurrent PPO (PPO LSTM), Truncated Quantile Critics (TQC), Quantile Regression DQN (QR-DQN) or PPO with invalid action masking (Maskable PPO).
|
||||||
|
|
||||||
Documentation is available online: [https://sb3-contrib.readthedocs.io/](https://sb3-contrib.readthedocs.io/)
|
Documentation is available online: [https://sb3-contrib.readthedocs.io/](https://sb3-contrib.readthedocs.io/)
|
||||||
|
|
||||||
|
|
@ -122,7 +122,7 @@ from stable_baselines3 import PPO
|
||||||
env = gym.make("CartPole-v1")
|
env = gym.make("CartPole-v1")
|
||||||
|
|
||||||
model = PPO("MlpPolicy", env, verbose=1)
|
model = PPO("MlpPolicy", env, verbose=1)
|
||||||
model.learn(total_timesteps=10000)
|
model.learn(total_timesteps=10_000)
|
||||||
|
|
||||||
obs = env.reset()
|
obs = env.reset()
|
||||||
for i in range(1000):
|
for i in range(1000):
|
||||||
|
|
@ -140,7 +140,7 @@ Or just train a model with a one liner if [the environment is registered in Gym]
|
||||||
```python
|
```python
|
||||||
from stable_baselines3 import PPO
|
from stable_baselines3 import PPO
|
||||||
|
|
||||||
model = PPO('MlpPolicy', 'CartPole-v1').learn(10000)
|
model = PPO("MlpPolicy", "CartPole-v1").learn(10_000)
|
||||||
```
|
```
|
||||||
|
|
||||||
Please read the [documentation](https://stable-baselines3.readthedocs.io/) for more examples.
|
Please read the [documentation](https://stable-baselines3.readthedocs.io/) for more examples.
|
||||||
|
|
@ -172,6 +172,7 @@ All the following examples can be executed online using Google colab notebooks:
|
||||||
| HER | :x: | :heavy_check_mark: | :heavy_check_mark: | :x: | :x: | :x: |
|
| HER | :x: | :heavy_check_mark: | :heavy_check_mark: | :x: | :x: | :x: |
|
||||||
| PPO | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
|
| PPO | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
|
||||||
| QR-DQN<sup>[1](#f1)</sup> | :x: | :x: | :heavy_check_mark: | :x: | :x: | :heavy_check_mark: |
|
| QR-DQN<sup>[1](#f1)</sup> | :x: | :x: | :heavy_check_mark: | :x: | :x: | :heavy_check_mark: |
|
||||||
|
| RecurrentPPO<sup>[1](#f1)</sup> | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
|
||||||
| SAC | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
|
| SAC | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
|
||||||
| TD3 | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
|
| TD3 | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
|
||||||
| TQC<sup>[1](#f1)</sup> | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
|
| TQC<sup>[1](#f1)</sup> | :x: | :heavy_check_mark: | :x: | :x: | :x: | :heavy_check_mark: |
|
||||||
|
|
|
||||||
|
|
@ -15,6 +15,7 @@ DQN ❌ ✔️ ❌ ❌
|
||||||
HER ✔️ ✔️ ❌ ❌ ❌
|
HER ✔️ ✔️ ❌ ❌ ❌
|
||||||
PPO ✔️ ✔️ ✔️ ✔️ ✔️
|
PPO ✔️ ✔️ ✔️ ✔️ ✔️
|
||||||
QR-DQN [#f1]_ ❌ ️ ✔️ ❌ ❌ ✔️
|
QR-DQN [#f1]_ ❌ ️ ✔️ ❌ ❌ ✔️
|
||||||
|
RecurrentPPO [#f1]_ ✔️ ✔️ ✔️ ✔️ ✔️
|
||||||
SAC ✔️ ❌ ❌ ❌ ✔️
|
SAC ✔️ ❌ ❌ ❌ ✔️
|
||||||
TD3 ✔️ ❌ ❌ ❌ ✔️
|
TD3 ✔️ ❌ ❌ ❌ ✔️
|
||||||
TQC [#f1]_ ✔️ ❌ ❌ ❌ ✔️
|
TQC [#f1]_ ✔️ ❌ ❌ ❌ ✔️
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,7 @@ We implement experimental features in a separate contrib repository:
|
||||||
`SB3-Contrib`_
|
`SB3-Contrib`_
|
||||||
|
|
||||||
This allows Stable-Baselines3 (SB3) to maintain a stable and compact core, while still
|
This allows Stable-Baselines3 (SB3) to maintain a stable and compact core, while still
|
||||||
providing the latest features, like Truncated Quantile Critics (TQC), Augmented Random Search (ARS), Trust Region Policy Optimization (TRPO) or
|
providing the latest features, like RecurrentPPO (PPO LSTM), Truncated Quantile Critics (TQC), Augmented Random Search (ARS), Trust Region Policy Optimization (TRPO) or
|
||||||
Quantile Regression DQN (QR-DQN).
|
Quantile Regression DQN (QR-DQN).
|
||||||
|
|
||||||
Why create this repository?
|
Why create this repository?
|
||||||
|
|
@ -38,9 +38,11 @@ See documentation for the full list of included features.
|
||||||
|
|
||||||
- `Augmented Random Search (ARS) <https://arxiv.org/abs/1803.07055>`_
|
- `Augmented Random Search (ARS) <https://arxiv.org/abs/1803.07055>`_
|
||||||
- `Quantile Regression DQN (QR-DQN)`_
|
- `Quantile Regression DQN (QR-DQN)`_
|
||||||
|
- `PPO with invalid action masking (Maskable PPO) <https://arxiv.org/abs/2006.14171>`_
|
||||||
|
- `PPO with recurrent policy (RecurrentPPO aka PPO LSTM) <https://ppo-details.cleanrl.dev//2021/11/05/ppo-implementation-details/>`_
|
||||||
- `Truncated Quantile Critics (TQC)`_
|
- `Truncated Quantile Critics (TQC)`_
|
||||||
- `Trust Region Policy Optimization (TRPO) <https://arxiv.org/abs/1502.05477>`_
|
- `Trust Region Policy Optimization (TRPO) <https://arxiv.org/abs/1502.05477>`_
|
||||||
- `PPO with invalid action masking (Maskable PPO) <https://arxiv.org/abs/2006.14171>`_
|
|
||||||
|
|
||||||
**Gym Wrappers**:
|
**Gym Wrappers**:
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ Changelog
|
||||||
==========
|
==========
|
||||||
|
|
||||||
|
|
||||||
Release 1.5.1a7 (WIP)
|
Release 1.5.1a8 (WIP)
|
||||||
---------------------------
|
---------------------------
|
||||||
|
|
||||||
Breaking Changes:
|
Breaking Changes:
|
||||||
|
|
@ -18,6 +18,8 @@ New Features:
|
||||||
|
|
||||||
SB3-Contrib
|
SB3-Contrib
|
||||||
^^^^^^^^^^^
|
^^^^^^^^^^^
|
||||||
|
- Added Recurrent PPO (PPO LSTM). See https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/pull/53
|
||||||
|
|
||||||
|
|
||||||
Bug Fixes:
|
Bug Fixes:
|
||||||
^^^^^^^^^^
|
^^^^^^^^^^
|
||||||
|
|
|
||||||
|
|
@ -15,7 +15,7 @@ For that, ppo uses clipping to avoid too large update.
|
||||||
.. note::
|
.. note::
|
||||||
|
|
||||||
PPO contains several modifications from the original algorithm not documented
|
PPO contains several modifications from the original algorithm not documented
|
||||||
by OpenAI: advantages are normalized and value function can be also clipped .
|
by OpenAI: advantages are normalized and value function can be also clipped.
|
||||||
|
|
||||||
|
|
||||||
Notes
|
Notes
|
||||||
|
|
@ -31,6 +31,16 @@ Notes
|
||||||
Can I use?
|
Can I use?
|
||||||
----------
|
----------
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
|
||||||
|
A recurrent version of PPO is available in our contrib repo: https://sb3-contrib.readthedocs.io/en/master/modules/ppo_recurrent.html
|
||||||
|
|
||||||
|
However we advise users to start with simple frame-stacking as a simpler, faster
|
||||||
|
and usually competitive alternative, more info in our report: https://wandb.ai/sb3/no-vel-envs/reports/PPO-vs-RecurrentPPO-aka-PPO-LSTM-on-environments-with-masked-velocity--VmlldzoxOTI4NjE4
|
||||||
|
See also `Procgen paper appendix Fig 11. <https://arxiv.org/abs/1912.01588>`_.
|
||||||
|
In practice, you can stack multiple observations using ``VecFrameStack``.
|
||||||
|
|
||||||
|
|
||||||
- Recurrent policies: ❌
|
- Recurrent policies: ❌
|
||||||
- Multi processing: ✔️
|
- Multi processing: ✔️
|
||||||
- Gym spaces:
|
- Gym spaces:
|
||||||
|
|
|
||||||
|
|
@ -1 +1 @@
|
||||||
1.5.1a7
|
1.5.1a8
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue