mirror of
https://github.com/saymrwulf/stable-baselines3.git
synced 2026-07-30 20:18:15 +00:00
Merge branch 'master' into sde
This commit is contained in:
Antonin RAFFIN
commit
e379a7a4c8
33 changed files with 497 additions and 202 deletions
8
.github/PULL_REQUEST_TEMPLATE.md
vendored
8
.github/PULL_REQUEST_TEMPLATE.md
vendored
|
|
@ -24,7 +24,11 @@
|
|||
- [ ] My change requires a change to the documentation.
|
||||
- [ ] I have updated the tests accordingly (*required for a bug fix or a new feature*).
|
||||
- [ ] I have updated the documentation accordingly.
|
||||
- [ ] I have checked the codestyle using `make lint`
|
||||
- [ ] I have ensured `make pytest` and `make type` both pass.
|
||||
- [ ] I have reformatted the code using `make format` (**required**)
|
||||
- [ ] I have checked the codestyle using `make check-codestyle` and `make lint` (**required**)
|
||||
- [ ] I have ensured `make pytest` and `make type` both pass. (**required**)
|
||||
|
||||
|
||||
Note: we are using a maximum length of 127 characters per line
|
||||
|
||||
<!--- This Template is an edited version of the one from https://github.com/evilsocket/pwnagotchi/ -->
|
||||
|
|
|
|||
3
.github/workflows/ci.yml
vendored
3
.github/workflows/ci.yml
vendored
|
|
@ -38,6 +38,9 @@ jobs:
|
|||
- name: Type check
|
||||
run: |
|
||||
make type
|
||||
- name: Check codestyle
|
||||
run: |
|
||||
make check-codestyle
|
||||
- name: Lint with flake8
|
||||
run: |
|
||||
make lint
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
image: stablebaselines/stable-baselines3-cpu:0.8.0a1
|
||||
image: stablebaselines/stable-baselines3-cpu:0.8.0a4
|
||||
|
||||
type-check:
|
||||
script:
|
||||
|
|
@ -15,4 +15,5 @@ doc-build:
|
|||
|
||||
lint-check:
|
||||
script:
|
||||
- make check-codestyle
|
||||
- make lint
|
||||
|
|
|
|||
|
|
@ -38,23 +38,9 @@ pip install -e .[docs,tests,extra]
|
|||
|
||||
## Codestyle
|
||||
|
||||
We follow the [PEP8 codestyle](https://www.python.org/dev/peps/pep-0008/). Please order the imports as follows:
|
||||
We are using [black codestyle](https://github.com/psf/black) (max line length of 127 characters) together with [isort](https://github.com/timothycrosley/isort) to sort the imports.
|
||||
|
||||
1. built-in
|
||||
2. packages
|
||||
3. current module
|
||||
|
||||
with one space between each, that gives for instance:
|
||||
```python
|
||||
import os
|
||||
import warnings
|
||||
|
||||
import numpy as np
|
||||
|
||||
from stable_baselines3 import PPO
|
||||
```
|
||||
|
||||
In general, we recommend using pycharm to format everything in an efficient way.
|
||||
**Please run `make format`** to reformat your code. You can check the codestyle using `make check-codestyle` and `make lint`.
|
||||
|
||||
Please document each function/method and [type](https://google.github.io/pytype/user_guide.html) them using the following template:
|
||||
|
||||
|
|
@ -77,11 +63,10 @@ def my_function(arg1: type1, arg2: type2) -> returntype:
|
|||
Before proposing a PR, please open an issue, where the feature will be discussed. This prevent from duplicated PR to be proposed and also ease the code review process.
|
||||
|
||||
Each PR need to be reviewed and accepted by at least one of the maintainers (@hill-a, @araffin, @erniejunior, @AdamGleave or @Miffyli).
|
||||
A PR must pass the Continuous Integration tests (travis + codacy) to be merged with the master branch.
|
||||
A PR must pass the Continuous Integration tests to be merged with the master branch.
|
||||
|
||||
Note: in rare cases, we can create exception for codacy failure.
|
||||
|
||||
## Test
|
||||
## Tests
|
||||
|
||||
All new features must add tests in the `tests/` folder ensuring that everything works fine.
|
||||
We use [pytest](https://pytest.org/).
|
||||
|
|
@ -99,12 +84,18 @@ Type checking with `pytype`:
|
|||
make type
|
||||
```
|
||||
|
||||
Codestyle check with `flake8`:
|
||||
Codestyle check with `black`, `isort` and `flake8`:
|
||||
|
||||
```
|
||||
make check-codestyle
|
||||
make lint
|
||||
```
|
||||
|
||||
To run `pytype`, `format` and `lint` in one command:
|
||||
```
|
||||
make commit-checks
|
||||
```
|
||||
|
||||
Build the documentation:
|
||||
|
||||
```
|
||||
|
|
@ -121,6 +112,7 @@ make spelling
|
|||
## Changelog and Documentation
|
||||
|
||||
Please do not forget to update the changelog (`docs/misc/changelog.rst`) and add documentation if needed.
|
||||
You should add your username next to each changelog entry that you added. If this is your first contribution, please add your username at the bottom too.
|
||||
A README is present in the `docs/` folder for instructions on how to build the documentation.
|
||||
|
||||
|
||||
|
|
|
|||
14
Dockerfile
14
Dockerfile
|
|
@ -10,7 +10,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
|
|||
curl \
|
||||
ca-certificates \
|
||||
libjpeg-dev \
|
||||
libpng-dev && \
|
||||
libpng-dev \
|
||||
libglib2.0-0 && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install anaconda abd dependencies
|
||||
|
|
@ -37,15 +38,4 @@ RUN \
|
|||
pip install opencv-python-headless && \
|
||||
rm -rf $HOME/.cache/pip
|
||||
|
||||
|
||||
# Codacy deps
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
default-jre \
|
||||
jq && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Codacy code coverage report: used for partial code coverage reporting
|
||||
RUN cd $CODE_DIR && \
|
||||
curl -Ls -o codacy-coverage-reporter.jar "$(curl -Ls https://api.github.com/repos/codacy/codacy-coverage-reporter/releases/latest | jq -r '.assets | map({name, browser_download_url} | select(.name | (startswith("codacy-coverage-reporter") and contains("assembly") and endswith(".jar")))) | .[0].browser_download_url')"
|
||||
|
||||
CMD /bin/bash
|
||||
|
|
|
|||
18
Makefile
18
Makefile
|
|
@ -14,6 +14,20 @@ lint:
|
|||
# exit-zero treats all errors as warnings.
|
||||
flake8 ${LINT_PATHS} --count --exit-zero --statistics
|
||||
|
||||
format:
|
||||
# Sort imports
|
||||
isort ${LINT_PATHS}
|
||||
# Reformat using black
|
||||
black -l 127 ${LINT_PATHS}
|
||||
|
||||
check-codestyle:
|
||||
# Sort imports
|
||||
isort --check ${LINT_PATHS}
|
||||
# Reformat using black
|
||||
black --check -l 127 ${LINT_PATHS}
|
||||
|
||||
commit-checks: format type lint
|
||||
|
||||
doc:
|
||||
cd docs && make html
|
||||
|
||||
|
|
@ -23,8 +37,6 @@ spelling:
|
|||
clean:
|
||||
cd docs && make clean
|
||||
|
||||
.PHONY: clean spelling doc lint
|
||||
|
||||
# Build docker images
|
||||
# If you do export RELEASE=True, it will also push them
|
||||
docker: docker-cpu docker-gpu
|
||||
|
|
@ -46,3 +58,5 @@ test-release:
|
|||
python setup.py sdist
|
||||
python setup.py bdist_wheel
|
||||
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
|
||||
|
||||
.PHONY: clean spelling doc lint format check-codestyle commit-checks
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
<img src="docs/\_static/img/logo.png" align="right" width="40%"/>
|
||||
|
||||
[](https://gitlab.com/araffin/stable-baselines3/-/commits/sde) [](https://stable-baselines3.readthedocs.io/en/master/?badge=master) [](https://gitlab.com/araffin/stable-baselines3/-/commits/sde)
|
||||
[](https://github.com/psf/black)
|
||||
|
||||
|
||||
# Generalized State-Dependent Exploration (gSDE) for Deep Reinforcement Learning in Robotics
|
||||
|
|
@ -46,7 +47,6 @@ These algorithms will make it easier for the research community and industry to
|
|||
Please look at the issue for more details.
|
||||
Planned features:
|
||||
|
||||
- [ ] DDPG (you can use its successor TD3 for now)
|
||||
- [ ] HER
|
||||
|
||||
### Planned features (v1.1+)
|
||||
|
|
@ -158,6 +158,7 @@ All the following examples can be executed online using Google colab notebooks:
|
|||
- [Monitor Training and Plotting](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/monitor_training.ipynb)
|
||||
- [Atari Games](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/atari_games.ipynb)
|
||||
- [RL Baselines Zoo](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/rl-baselines-zoo.ipynb)
|
||||
- [PyBullet](https://colab.research.google.com/github/Stable-Baselines-Team/rl-colab-notebooks/blob/sb3/pybullet.ipynb)
|
||||
|
||||
|
||||
## Implemented Algorithms
|
||||
|
|
@ -165,6 +166,8 @@ All the following examples can be executed online using Google colab notebooks:
|
|||
| **Name** | **Recurrent** | `Box` | `Discrete` | `MultiDiscrete` | `MultiBinary` | **Multi Processing** |
|
||||
| ------------------- | ------------------ | ------------------ | ------------------ | ------------------- | ------------------ | --------------------------------- |
|
||||
| A2C | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
|
||||
| DDPG | :x: | :heavy_check_mark: | :x: | :x: | :x: | :x: |
|
||||
| DQN | :x: | :x: | :heavy_check_mark: | :x: | :x: | :x: |
|
||||
| PPO | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: |
|
||||
| SAC | :x: | :heavy_check_mark: | :x: | :x: | :x: | :x: |
|
||||
| TD3 | :x: | :heavy_check_mark: | :x: | :x: | :x: | :x: |
|
||||
|
|
|
|||
72
docs/conf.py
72
docs/conf.py
|
|
@ -20,12 +20,13 @@ from unittest.mock import MagicMock
|
|||
# PyEnchant.
|
||||
try:
|
||||
import sphinxcontrib.spelling # noqa: F401
|
||||
|
||||
enable_spell_check = True
|
||||
except ImportError:
|
||||
enable_spell_check = False
|
||||
|
||||
# source code directory, relative to this file, for sphinx-autobuild
|
||||
sys.path.insert(0, os.path.abspath('..'))
|
||||
sys.path.insert(0, os.path.abspath(".."))
|
||||
|
||||
|
||||
class Mock(MagicMock):
|
||||
|
|
@ -44,18 +45,18 @@ MOCK_MODULES = []
|
|||
sys.modules.update((mod_name, Mock()) for mod_name in MOCK_MODULES)
|
||||
|
||||
# Read version from file
|
||||
version_file = os.path.join(os.path.dirname(__file__), '../stable_baselines3', 'version.txt')
|
||||
with open(version_file, 'r') as file_handler:
|
||||
version_file = os.path.join(os.path.dirname(__file__), "../stable_baselines3", "version.txt")
|
||||
with open(version_file, "r") as file_handler:
|
||||
__version__ = file_handler.read().strip()
|
||||
|
||||
# -- Project information -----------------------------------------------------
|
||||
|
||||
project = 'Stable Baselines3'
|
||||
copyright = '2020, Stable Baselines3'
|
||||
author = 'Stable Baselines3 Contributors'
|
||||
project = "Stable Baselines3"
|
||||
copyright = "2020, Stable Baselines3"
|
||||
author = "Stable Baselines3 Contributors"
|
||||
|
||||
# The short X.Y version
|
||||
version = 'master (' + __version__ + ' )'
|
||||
version = "master (" + __version__ + " )"
|
||||
# The full version, including alpha/beta/rc tags
|
||||
release = __version__
|
||||
|
||||
|
|
@ -70,30 +71,30 @@ release = __version__
|
|||
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
|
||||
# ones.
|
||||
extensions = [
|
||||
'sphinx.ext.autodoc',
|
||||
"sphinx.ext.autodoc",
|
||||
# 'sphinx_autodoc_typehints',
|
||||
'sphinx.ext.autosummary',
|
||||
'sphinx.ext.mathjax',
|
||||
'sphinx.ext.ifconfig',
|
||||
'sphinx.ext.viewcode',
|
||||
"sphinx.ext.autosummary",
|
||||
"sphinx.ext.mathjax",
|
||||
"sphinx.ext.ifconfig",
|
||||
"sphinx.ext.viewcode",
|
||||
# 'sphinx.ext.intersphinx',
|
||||
# 'sphinx.ext.doctest'
|
||||
]
|
||||
|
||||
if enable_spell_check:
|
||||
extensions.append('sphinxcontrib.spelling')
|
||||
extensions.append("sphinxcontrib.spelling")
|
||||
|
||||
# Add any paths that contain templates here, relative to this directory.
|
||||
templates_path = ['_templates']
|
||||
templates_path = ["_templates"]
|
||||
|
||||
# The suffix(es) of source filenames.
|
||||
# You can specify multiple suffix as a list of string:
|
||||
#
|
||||
# source_suffix = ['.rst', '.md']
|
||||
source_suffix = '.rst'
|
||||
source_suffix = ".rst"
|
||||
|
||||
# The master toctree document.
|
||||
master_doc = 'index'
|
||||
master_doc = "index"
|
||||
|
||||
# The language for content autogenerated by Sphinx. Refer to documentation
|
||||
# for a list of supported languages.
|
||||
|
|
@ -105,10 +106,10 @@ language = None
|
|||
# List of patterns, relative to source directory, that match files and
|
||||
# directories to ignore when looking for source files.
|
||||
# This pattern also affects html_static_path and html_extra_path .
|
||||
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
|
||||
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
|
||||
|
||||
# The name of the Pygments (syntax highlighting) style to use.
|
||||
pygments_style = 'sphinx'
|
||||
pygments_style = "sphinx"
|
||||
|
||||
|
||||
# -- Options for HTML output -------------------------------------------------
|
||||
|
|
@ -117,13 +118,13 @@ pygments_style = 'sphinx'
|
|||
# a list of builtin themes.
|
||||
|
||||
# Fix for read the docs
|
||||
on_rtd = os.environ.get('READTHEDOCS') == 'True'
|
||||
on_rtd = os.environ.get("READTHEDOCS") == "True"
|
||||
if on_rtd:
|
||||
html_theme = 'default'
|
||||
html_theme = "default"
|
||||
else:
|
||||
html_theme = 'sphinx_rtd_theme'
|
||||
html_theme = "sphinx_rtd_theme"
|
||||
|
||||
html_logo = '_static/img/logo.png'
|
||||
html_logo = "_static/img/logo.png"
|
||||
|
||||
|
||||
def setup(app):
|
||||
|
|
@ -139,7 +140,7 @@ def setup(app):
|
|||
# Add any paths that contain custom static files (such as style sheets) here,
|
||||
# relative to this directory. They are copied after the builtin static files,
|
||||
# so a file named "default.css" will overwrite the builtin "default.css".
|
||||
html_static_path = ['_static']
|
||||
html_static_path = ["_static"]
|
||||
|
||||
# Custom sidebar templates, must be a dictionary that maps document names
|
||||
# to template names.
|
||||
|
|
@ -155,7 +156,7 @@ html_static_path = ['_static']
|
|||
# -- Options for HTMLHelp output ---------------------------------------------
|
||||
|
||||
# Output file base name for HTML help builder.
|
||||
htmlhelp_basename = 'StableBaselines3doc'
|
||||
htmlhelp_basename = "StableBaselines3doc"
|
||||
|
||||
|
||||
# -- Options for LaTeX output ------------------------------------------------
|
||||
|
|
@ -164,15 +165,12 @@ latex_elements = {
|
|||
# The paper size ('letterpaper' or 'a4paper').
|
||||
#
|
||||
# 'papersize': 'letterpaper',
|
||||
|
||||
# The font size ('10pt', '11pt' or '12pt').
|
||||
#
|
||||
# 'pointsize': '10pt',
|
||||
|
||||
# Additional stuff for the LaTeX preamble.
|
||||
#
|
||||
# 'preamble': '',
|
||||
|
||||
# Latex figure (float) alignment
|
||||
#
|
||||
# 'figure_align': 'htbp',
|
||||
|
|
@ -182,8 +180,7 @@ latex_elements = {
|
|||
# (source start file, target name, title,
|
||||
# author, documentclass [howto, manual, or own class]).
|
||||
latex_documents = [
|
||||
(master_doc, 'StableBaselines3.tex', 'Stable Baselines3 Documentation',
|
||||
'Stable Baselines3 Contributors', 'manual'),
|
||||
(master_doc, "StableBaselines3.tex", "Stable Baselines3 Documentation", "Stable Baselines3 Contributors", "manual"),
|
||||
]
|
||||
|
||||
|
||||
|
|
@ -191,10 +188,7 @@ latex_documents = [
|
|||
|
||||
# One entry per manual page. List of tuples
|
||||
# (source start file, name, description, authors, manual section).
|
||||
man_pages = [
|
||||
(master_doc, 'stablebaselines3', 'Stable Baselines3 Documentation',
|
||||
[author], 1)
|
||||
]
|
||||
man_pages = [(master_doc, "stablebaselines3", "Stable Baselines3 Documentation", [author], 1)]
|
||||
|
||||
|
||||
# -- Options for Texinfo output ----------------------------------------------
|
||||
|
|
@ -203,9 +197,15 @@ man_pages = [
|
|||
# (source start file, target name, title, author,
|
||||
# dir menu entry, description, category)
|
||||
texinfo_documents = [
|
||||
(master_doc, 'StableBaselines3', 'Stable Baselines3 Documentation',
|
||||
author, 'StableBaselines3', 'One line description of project.',
|
||||
'Miscellaneous'),
|
||||
(
|
||||
master_doc,
|
||||
"StableBaselines3",
|
||||
"Stable Baselines3 Documentation",
|
||||
author,
|
||||
"StableBaselines3",
|
||||
"One line description of project.",
|
||||
"Miscellaneous",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -9,10 +9,11 @@ along with some useful characteristics: support for discrete/continuous actions,
|
|||
Name ``Box`` ``Discrete`` ``MultiDiscrete`` ``MultiBinary`` Multi Processing
|
||||
============ =========== ============ ================= =============== ================
|
||||
A2C ✔️ ✔️ ✔️ ✔️ ✔️
|
||||
DDPG ✔️ ❌ ❌ ❌ ❌
|
||||
DQN ❌ ✔️ ❌ ❌ ❌
|
||||
PPO ✔️ ✔️ ✔️ ✔️ ✔️
|
||||
SAC ✔️ ❌ ❌ ❌ ❌
|
||||
TD3 ✔️ ❌ ❌ ❌ ❌
|
||||
DQN ❌ ✔️ ❌ ❌ ❌
|
||||
============ =========== ============ ================= =============== ================
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -37,8 +37,8 @@ You can also easily define a custom architecture for the policy (or value) netwo
|
|||
.. note::
|
||||
|
||||
Defining a custom policy class is equivalent to passing ``policy_kwargs``.
|
||||
However, it lets you name the policy and so makes usually the code clearer.
|
||||
``policy_kwargs`` should be rather used when doing hyperparameter search.
|
||||
However, it lets you name the policy and so usually makes the code clearer.
|
||||
``policy_kwargs`` is particularly useful when doing hyperparameter search.
|
||||
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -55,10 +55,11 @@ Main Features
|
|||
|
||||
modules/base
|
||||
modules/a2c
|
||||
modules/ddpg
|
||||
modules/dqn
|
||||
modules/ppo
|
||||
modules/sac
|
||||
modules/td3
|
||||
modules/dqn
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
Changelog
|
||||
==========
|
||||
|
||||
Pre-Release 0.8.0a3 (WIP)
|
||||
Pre-Release 0.8.0a4 (WIP)
|
||||
------------------------------
|
||||
|
||||
Breaking Changes:
|
||||
|
|
@ -12,6 +12,8 @@ Breaking Changes:
|
|||
- ``save_replay_buffer`` now receives as argument the file path instead of the folder path (@tirafesi)
|
||||
- Refactored ``Critic`` class for ``TD3`` and ``SAC``, it is now called ``ContinuousCritic``
|
||||
and has an additional parameter ``n_critics``
|
||||
- ``SAC`` and ``TD3`` now accept an arbitrary number of critics (e.g. ``policy_kwargs=dict(n_critics=3)``)
|
||||
instead of only 2 previously
|
||||
|
||||
New Features:
|
||||
^^^^^^^^^^^^^
|
||||
|
|
@ -21,6 +23,7 @@ New Features:
|
|||
when ``psutil`` is available
|
||||
- Saving models now automatically creates the necessary folders and raises appropriate warnings (@PartiallyTyped)
|
||||
- Refactored opening paths for saving and loading to use strings, pathlib or io.BufferedIOBase (@PartiallyTyped)
|
||||
- Added ``DDPG`` algorithm as a special case of ``TD3``.
|
||||
- Introduced ``BaseModel`` abstract parent for ``BasePolicy``, which critics inherit from.
|
||||
|
||||
Bug Fixes:
|
||||
|
|
@ -37,7 +40,10 @@ Others:
|
|||
- Split the ``collect_rollout()`` method for off-policy algorithms
|
||||
- Added ``_on_step()`` for off-policy base class
|
||||
- Optimized replay buffer size by removing the need of ``next_observations`` numpy array
|
||||
- Switch to ``black`` codestyle and added ``make format``, ``make check-codestyle`` and ``commit-checks``
|
||||
- Ignored errors from newer pytype version
|
||||
- Added a check when using ``gSDE``
|
||||
- Removed codacy dependency from Dockerfile
|
||||
|
||||
Documentation:
|
||||
^^^^^^^^^^^^^^
|
||||
|
|
|
|||
104
docs/modules/ddpg.rst
Normal file
104
docs/modules/ddpg.rst
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
.. _ddpg:
|
||||
|
||||
.. automodule:: stable_baselines3.ddpg
|
||||
|
||||
|
||||
DDPG
|
||||
====
|
||||
|
||||
`Deep Deterministic Policy Gradient (DDPG) <https://spinningup.openai.com/en/latest/algorithms/ddpg.html>`_ combines the
|
||||
trick for DQN with the deterministic policy gradient, to obtain an algorithm for continuous actions.
|
||||
|
||||
|
||||
.. rubric:: Available Policies
|
||||
|
||||
.. autosummary::
|
||||
:nosignatures:
|
||||
|
||||
MlpPolicy
|
||||
|
||||
|
||||
Notes
|
||||
-----
|
||||
|
||||
- Deterministic Policy Gradient: http://proceedings.mlr.press/v32/silver14.pdf
|
||||
- DDPG Paper: https://arxiv.org/abs/1509.02971
|
||||
- OpenAI Spinning Guide for DDPG: https://spinningup.openai.com/en/latest/algorithms/ddpg.html
|
||||
|
||||
.. note::
|
||||
|
||||
The default policy for DDPG uses a ReLU activation, to match the original paper, whereas most other algorithms' MlpPolicy uses a tanh activation.
|
||||
to match the original paper
|
||||
|
||||
|
||||
Can I use?
|
||||
----------
|
||||
|
||||
- Recurrent policies: ❌
|
||||
- Multi processing: ❌
|
||||
- Gym spaces:
|
||||
|
||||
|
||||
============= ====== ===========
|
||||
Space Action Observation
|
||||
============= ====== ===========
|
||||
Discrete ❌ ✔️
|
||||
Box ✔️ ✔️
|
||||
MultiDiscrete ❌ ✔️
|
||||
MultiBinary ❌ ✔️
|
||||
============= ====== ===========
|
||||
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import gym
|
||||
import numpy as np
|
||||
|
||||
from stable_baselines3 import DDPG
|
||||
from stable_baselines3.common.noise import NormalActionNoise, OrnsteinUhlenbeckActionNoise
|
||||
|
||||
env = gym.make('Pendulum-v0')
|
||||
|
||||
# The noise objects for DDPG
|
||||
n_actions = env.action_space.shape[-1]
|
||||
action_noise = NormalActionNoise(mean=np.zeros(n_actions), sigma=0.1 * np.ones(n_actions))
|
||||
|
||||
model = DDPG('MlpPolicy', env, action_noise=action_noise, verbose=1)
|
||||
model.learn(total_timesteps=10000, log_interval=10)
|
||||
model.save("ddpg_pendulum")
|
||||
env = model.get_env()
|
||||
|
||||
del model # remove to demonstrate saving and loading
|
||||
|
||||
model = DDPG.load("ddpg_pendulum")
|
||||
|
||||
obs = env.reset()
|
||||
while True:
|
||||
action, _states = model.predict(obs)
|
||||
obs, rewards, dones, info = env.step(action)
|
||||
env.render()
|
||||
|
||||
|
||||
Parameters
|
||||
----------
|
||||
|
||||
.. autoclass:: DDPG
|
||||
:members:
|
||||
:inherited-members:
|
||||
|
||||
.. _ddpg_policies:
|
||||
|
||||
DDPG Policies
|
||||
-------------
|
||||
|
||||
.. autoclass:: MlpPolicy
|
||||
:members:
|
||||
:inherited-members:
|
||||
|
||||
|
||||
.. .. autoclass:: CnnPolicy
|
||||
.. :members:
|
||||
.. :inherited-members:
|
||||
|
|
@ -117,3 +117,5 @@ Deprecations
|
|||
forkserver
|
||||
cuda
|
||||
Polyak
|
||||
gSDE
|
||||
rollouts
|
||||
|
|
|
|||
|
|
@ -21,12 +21,13 @@ filterwarnings =
|
|||
inputs = stable_baselines3
|
||||
|
||||
[flake8]
|
||||
ignore = W503,W504 # line breaks before and after binary operators
|
||||
ignore = W503,W504,E203,E231 # line breaks before and after binary operators
|
||||
# Ignore import not used when aliases are defined
|
||||
per-file-ignores =
|
||||
./stable_baselines3/__init__.py:F401
|
||||
./stable_baselines3/common/__init__.py:F401
|
||||
./stable_baselines3/a2c/__init__.py:F401
|
||||
./stable_baselines3/ddpg/__init__.py:F401
|
||||
./stable_baselines3/dqn/__init__.py:F401
|
||||
./stable_baselines3/ppo/__init__.py:F401
|
||||
./stable_baselines3/sac/__init__.py:F401
|
||||
|
|
@ -48,7 +49,6 @@ max-complexity = 15
|
|||
# The GitHub editor is 127 chars wide
|
||||
max-line-length = 127
|
||||
|
||||
|
||||
[isort]
|
||||
profile = black
|
||||
line_length = 127
|
||||
|
|
|
|||
128
setup.py
128
setup.py
|
|
@ -1,7 +1,8 @@
|
|||
import os
|
||||
from setuptools import setup, find_packages
|
||||
|
||||
with open(os.path.join('stable_baselines3', 'version.txt'), 'r') as file_handler:
|
||||
from setuptools import find_packages, setup
|
||||
|
||||
with open(os.path.join("stable_baselines3", "version.txt"), "r") as file_handler:
|
||||
__version__ = file_handler.read().strip()
|
||||
|
||||
|
||||
|
|
@ -64,66 +65,69 @@ model = PPO('MlpPolicy', 'CartPole-v1').learn(10000)
|
|||
""" # noqa:E501
|
||||
|
||||
|
||||
setup(name='stable_baselines3',
|
||||
packages=[package for package in find_packages()
|
||||
if package.startswith('stable_baselines3')],
|
||||
package_data={
|
||||
'stable_baselines3': ['py.typed', 'version.txt']
|
||||
},
|
||||
install_requires=[
|
||||
'gym>=0.17',
|
||||
'numpy',
|
||||
'torch>=1.4.0',
|
||||
# For saving models
|
||||
'cloudpickle',
|
||||
# For reading logs
|
||||
'pandas',
|
||||
# Plotting learning curves
|
||||
'matplotlib'
|
||||
],
|
||||
extras_require={
|
||||
'tests': [
|
||||
# Run tests and coverage
|
||||
'pytest',
|
||||
'pytest-cov',
|
||||
'pytest-env',
|
||||
'pytest-xdist',
|
||||
# Type check
|
||||
'pytype',
|
||||
# Lint code
|
||||
'flake8>=3.8'
|
||||
],
|
||||
'docs': [
|
||||
'sphinx',
|
||||
'sphinx-autobuild',
|
||||
'sphinx-rtd-theme',
|
||||
# For spelling
|
||||
'sphinxcontrib.spelling',
|
||||
# Type hints support
|
||||
# 'sphinx-autodoc-typehints'
|
||||
],
|
||||
'extra': [
|
||||
# For render
|
||||
'opencv-python',
|
||||
# For atari games,
|
||||
'atari_py~=0.2.0', 'pillow',
|
||||
# Tensorboard support
|
||||
'tensorboard',
|
||||
# Checking memory taken by replay buffer
|
||||
'psutil'
|
||||
]
|
||||
},
|
||||
description='Pytorch version of Stable Baselines, implementations of reinforcement learning algorithms.',
|
||||
author='Antonin Raffin',
|
||||
url='https://github.com/DLR-RM/stable-baselines3',
|
||||
author_email='antonin.raffin@dlr.de',
|
||||
keywords="reinforcement-learning-algorithms reinforcement-learning machine-learning "
|
||||
"gym openai stable baselines toolbox python data-science",
|
||||
license="MIT",
|
||||
long_description=long_description,
|
||||
long_description_content_type='text/markdown',
|
||||
version=__version__,
|
||||
)
|
||||
setup(
|
||||
name="stable_baselines3",
|
||||
packages=[package for package in find_packages() if package.startswith("stable_baselines3")],
|
||||
package_data={"stable_baselines3": ["py.typed", "version.txt"]},
|
||||
install_requires=[
|
||||
"gym>=0.17",
|
||||
"numpy",
|
||||
"torch>=1.4.0",
|
||||
# For saving models
|
||||
"cloudpickle",
|
||||
# For reading logs
|
||||
"pandas",
|
||||
# Plotting learning curves
|
||||
"matplotlib",
|
||||
],
|
||||
extras_require={
|
||||
"tests": [
|
||||
# Run tests and coverage
|
||||
"pytest",
|
||||
"pytest-cov",
|
||||
"pytest-env",
|
||||
"pytest-xdist",
|
||||
# Type check
|
||||
"pytype",
|
||||
# Lint code
|
||||
"flake8>=3.8",
|
||||
# Sort imports
|
||||
"isort>=5.0",
|
||||
# Reformat
|
||||
"black",
|
||||
],
|
||||
"docs": [
|
||||
"sphinx",
|
||||
"sphinx-autobuild",
|
||||
"sphinx-rtd-theme",
|
||||
# For spelling
|
||||
"sphinxcontrib.spelling",
|
||||
# Type hints support
|
||||
# 'sphinx-autodoc-typehints'
|
||||
],
|
||||
"extra": [
|
||||
# For render
|
||||
"opencv-python",
|
||||
# For atari games,
|
||||
"atari_py~=0.2.0",
|
||||
"pillow",
|
||||
# Tensorboard support
|
||||
"tensorboard",
|
||||
# Checking memory taken by replay buffer
|
||||
"psutil",
|
||||
],
|
||||
},
|
||||
description="Pytorch version of Stable Baselines, implementations of reinforcement learning algorithms.",
|
||||
author="Antonin Raffin",
|
||||
url="https://github.com/DLR-RM/stable-baselines3",
|
||||
author_email="antonin.raffin@dlr.de",
|
||||
keywords="reinforcement-learning-algorithms reinforcement-learning machine-learning "
|
||||
"gym openai stable baselines toolbox python data-science",
|
||||
license="MIT",
|
||||
long_description=long_description,
|
||||
long_description_content_type="text/markdown",
|
||||
version=__version__,
|
||||
)
|
||||
|
||||
# python setup.py sdist
|
||||
# python setup.py bdist_wheel
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
import os
|
||||
|
||||
from stable_baselines3.a2c import A2C
|
||||
from stable_baselines3.ddpg import DDPG
|
||||
from stable_baselines3.dqn import DQN
|
||||
from stable_baselines3.ppo import PPO
|
||||
from stable_baselines3.sac import SAC
|
||||
|
|
|
|||
|
|
@ -157,6 +157,9 @@ class BaseAlgorithm(ABC):
|
|||
"Error: the model does not support multiple envs; it requires " "a single vectorized environment."
|
||||
)
|
||||
|
||||
if self.use_sde and not isinstance(self.observation_space, gym.spaces.Box):
|
||||
raise ValueError("generalized State-Dependent Exploration (gSDE) can only be used with continuous actions.")
|
||||
|
||||
def _wrap_env(self, env: GymEnv) -> VecEnv:
|
||||
if not isinstance(env, VecEnv):
|
||||
if self.verbose >= 1:
|
||||
|
|
@ -346,8 +349,11 @@ class BaseAlgorithm(ABC):
|
|||
|
||||
# noinspection PyArgumentList
|
||||
model = cls(
|
||||
policy=data["policy_class"], env=env, device="auto", _init_setup_model=False
|
||||
) # pytype: disable=not-instantiable,wrong-keyword-args
|
||||
policy=data["policy_class"],
|
||||
env=env,
|
||||
device="auto",
|
||||
_init_setup_model=False, # pytype: disable=not-instantiable,wrong-keyword-args
|
||||
)
|
||||
|
||||
# load parameters
|
||||
model.__dict__.update(data)
|
||||
|
|
|
|||
|
|
@ -82,19 +82,19 @@ def _check_obs(obs: Union[tuple, dict, np.ndarray, int], observation_space: spac
|
|||
if not isinstance(observation_space, spaces.Tuple):
|
||||
assert not isinstance(
|
||||
obs, tuple
|
||||
), "The observation returned by the `{}()` " "method should be a single value, not a tuple".format(method_name)
|
||||
), "The observation returned by the `{}()` method should be a single value, not a tuple".format(method_name)
|
||||
|
||||
# The check for a GoalEnv is done by the base class
|
||||
if isinstance(observation_space, spaces.Discrete):
|
||||
assert isinstance(obs, int), "The observation returned by `{}()` method must be an int".format(method_name)
|
||||
elif _enforce_array_obs(observation_space):
|
||||
assert isinstance(obs, np.ndarray), "The observation returned by `{}()` " "method must be a numpy array".format(
|
||||
assert isinstance(obs, np.ndarray), "The observation returned by `{}()` method must be a numpy array".format(
|
||||
method_name
|
||||
)
|
||||
|
||||
assert observation_space.contains(
|
||||
obs
|
||||
), "The observation returned by the `{}()` " "method does not match the given observation space".format(method_name)
|
||||
), "The observation returned by the `{}()` method does not match the given observation space".format(method_name)
|
||||
|
||||
|
||||
def _check_returned_values(env: gym.Env, observation_space: spaces.Space, action_space: spaces.Space) -> None:
|
||||
|
|
@ -187,9 +187,9 @@ def check_env(env: gym.Env, warn: bool = True, skip_render_check: bool = True) -
|
|||
:param skip_render_check: (bool) Whether to skip the checks for the render method.
|
||||
True by default (useful for the CI)
|
||||
"""
|
||||
assert isinstance(env, gym.Env), (
|
||||
"You environment must inherit from gym.Env class " " cf https://github.com/openai/gym/blob/master/gym/core.py"
|
||||
)
|
||||
assert isinstance(
|
||||
env, gym.Env
|
||||
), "You environment must inherit from gym.Env class cf https://github.com/openai/gym/blob/master/gym/core.py"
|
||||
|
||||
# ============= Check the spaces (observation and action) ================
|
||||
_check_spaces(env)
|
||||
|
|
|
|||
|
|
@ -46,7 +46,7 @@ class NormalActionNoise(ActionNoise):
|
|||
|
||||
class OrnsteinUhlenbeckActionNoise(ActionNoise):
|
||||
"""
|
||||
An Ornstein Uhlenbeck action noise, this is designed to aproximate brownian motion with friction.
|
||||
An Ornstein Uhlenbeck action noise, this is designed to approximate Brownian motion with friction.
|
||||
|
||||
Based on http://math.stackexchange.com/questions/1287634/implementing-ornstein-uhlenbeck-in-matlab
|
||||
|
||||
|
|
|
|||
|
|
@ -35,10 +35,13 @@ class OffPolicyAlgorithm(BaseAlgorithm):
|
|||
:param batch_size: (int) Minibatch size for each gradient update
|
||||
:param tau: (float) the soft update coefficient ("Polyak update", between 0 and 1)
|
||||
:param gamma: (float) the discount factor
|
||||
:param train_freq: (int) Update the model every ``train_freq`` steps.
|
||||
:param gradient_steps: (int) How many gradient update after each step
|
||||
:param train_freq: (int) Update the model every ``train_freq`` steps. Set to `-1` to disable.
|
||||
:param gradient_steps: (int) How many gradient steps to do after each rollout
|
||||
(see ``train_freq`` and ``n_episodes_rollout``)
|
||||
Set to ``-1`` means to do as many gradient steps as steps done in the environment
|
||||
during the rollout.
|
||||
:param n_episodes_rollout: (int) Update the model every ``n_episodes_rollout`` episodes.
|
||||
Note that this cannot be used at the same time as ``train_freq``
|
||||
Note that this cannot be used at the same time as ``train_freq``. Set to `-1` to disable.
|
||||
:param action_noise: (ActionNoise) the action noise type (None by default), this can help
|
||||
for hard exploration problem. Cf common.noise for the different action noise type.
|
||||
:param optimize_memory_usage: (bool) Enable a memory efficient variant of the replay buffer
|
||||
|
|
@ -153,8 +156,11 @@ class OffPolicyAlgorithm(BaseAlgorithm):
|
|||
optimize_memory_usage=self.optimize_memory_usage,
|
||||
)
|
||||
self.policy = self.policy_class(
|
||||
self.observation_space, self.action_space, self.lr_schedule, **self.policy_kwargs
|
||||
) # pytype:disable=not-instantiable
|
||||
self.observation_space,
|
||||
self.action_space,
|
||||
self.lr_schedule,
|
||||
**self.policy_kwargs # pytype:disable=not-instantiable
|
||||
)
|
||||
self.policy = self.policy.to(self.device)
|
||||
|
||||
def save_replay_buffer(self, path: Union[str, pathlib.Path, io.BufferedIOBase]) -> None:
|
||||
|
|
|
|||
|
|
@ -117,8 +117,8 @@ class OnPolicyAlgorithm(BaseAlgorithm):
|
|||
self.lr_schedule,
|
||||
use_sde=self.use_sde,
|
||||
device=self.device,
|
||||
**self.policy_kwargs
|
||||
) # pytype:disable=not-instantiable
|
||||
**self.policy_kwargs # pytype:disable=not-instantiable
|
||||
)
|
||||
self.policy = self.policy.to(self.device)
|
||||
|
||||
def collect_rollouts(
|
||||
|
|
|
|||
2
stable_baselines3/ddpg/__init__.py
Normal file
2
stable_baselines3/ddpg/__init__.py
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
from stable_baselines3.ddpg.ddpg import DDPG
|
||||
from stable_baselines3.ddpg.policies import CnnPolicy, MlpPolicy
|
||||
136
stable_baselines3/ddpg/ddpg.py
Normal file
136
stable_baselines3/ddpg/ddpg.py
Normal file
|
|
@ -0,0 +1,136 @@
|
|||
from typing import Any, Callable, Dict, Optional, Type, Union
|
||||
|
||||
import torch as th
|
||||
|
||||
from stable_baselines3.common.noise import ActionNoise
|
||||
from stable_baselines3.common.off_policy_algorithm import OffPolicyAlgorithm
|
||||
from stable_baselines3.common.type_aliases import GymEnv, MaybeCallback
|
||||
from stable_baselines3.td3.policies import TD3Policy
|
||||
from stable_baselines3.td3.td3 import TD3
|
||||
|
||||
|
||||
class DDPG(TD3):
|
||||
"""
|
||||
Deep Deterministic Policy Gradient (DDPG).
|
||||
|
||||
Deterministic Policy Gradient: http://proceedings.mlr.press/v32/silver14.pdf
|
||||
DDPG Paper: https://arxiv.org/abs/1509.02971
|
||||
Introduction to DDPG: https://spinningup.openai.com/en/latest/algorithms/ddpg.html
|
||||
|
||||
Note: we treat DDPG as a special case of its successor TD3.
|
||||
|
||||
:param policy: (DDPGPolicy or str) The policy model to use (MlpPolicy, CnnPolicy, ...)
|
||||
:param env: (GymEnv or str) The environment to learn from (if registered in Gym, can be str)
|
||||
:param learning_rate: (float or callable) learning rate for adam optimizer,
|
||||
the same learning rate will be used for all networks (Q-Values, Actor and Value function)
|
||||
it can be a function of the current progress remaining (from 1 to 0)
|
||||
:param buffer_size: (int) size of the replay buffer
|
||||
:param learning_starts: (int) how many steps of the model to collect transitions for before learning starts
|
||||
:param batch_size: (int) Minibatch size for each gradient update
|
||||
:param tau: (float) the soft update coefficient ("Polyak update", between 0 and 1)
|
||||
:param gamma: (float) the discount factor
|
||||
:param train_freq: (int) Update the model every ``train_freq`` steps. Set to `-1` to disable.
|
||||
:param gradient_steps: (int) How many gradient steps to do after each rollout
|
||||
(see ``train_freq`` and ``n_episodes_rollout``)
|
||||
Set to ``-1`` means to do as many gradient steps as steps done in the environment
|
||||
during the rollout.
|
||||
:param n_episodes_rollout: (int) Update the model every ``n_episodes_rollout`` episodes.
|
||||
Note that this cannot be used at the same time as ``train_freq``. Set to `-1` to disable.
|
||||
:param action_noise: (ActionNoise) the action noise type (None by default), this can help
|
||||
for hard exploration problem. Cf common.noise for the different action noise type.
|
||||
:param optimize_memory_usage: (bool) Enable a memory efficient variant of the replay buffer
|
||||
at a cost of more complexity.
|
||||
See https://github.com/DLR-RM/stable-baselines3/issues/37#issuecomment-637501195
|
||||
:param create_eval_env: (bool) Whether to create a second environment that will be
|
||||
used for evaluating the agent periodically. (Only available when passing string for the environment)
|
||||
:param policy_kwargs: (dict) additional arguments to be passed to the policy on creation
|
||||
:param verbose: (int) the verbosity level: 0 no output, 1 info, 2 debug
|
||||
:param seed: (int) Seed for the pseudo random generators
|
||||
:param device: (str or th.device) Device (cpu, cuda, ...) on which the code should be run.
|
||||
Setting it to auto, the code will be run on the GPU if possible.
|
||||
:param _init_setup_model: (bool) Whether or not to build the network at the creation of the instance
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
policy: Union[str, Type[TD3Policy]],
|
||||
env: Union[GymEnv, str],
|
||||
learning_rate: Union[float, Callable] = 1e-3,
|
||||
buffer_size: int = int(1e6),
|
||||
learning_starts: int = 100,
|
||||
batch_size: int = 100,
|
||||
tau: float = 0.005,
|
||||
gamma: float = 0.99,
|
||||
train_freq: int = -1,
|
||||
gradient_steps: int = -1,
|
||||
n_episodes_rollout: int = 1,
|
||||
action_noise: Optional[ActionNoise] = None,
|
||||
optimize_memory_usage: bool = False,
|
||||
tensorboard_log: Optional[str] = None,
|
||||
create_eval_env: bool = False,
|
||||
policy_kwargs: Dict[str, Any] = None,
|
||||
verbose: int = 0,
|
||||
seed: Optional[int] = None,
|
||||
device: Union[th.device, str] = "auto",
|
||||
_init_setup_model: bool = True,
|
||||
):
|
||||
|
||||
super(DDPG, self).__init__(
|
||||
policy=policy,
|
||||
env=env,
|
||||
learning_rate=learning_rate,
|
||||
buffer_size=buffer_size,
|
||||
learning_starts=learning_starts,
|
||||
batch_size=batch_size,
|
||||
tau=tau,
|
||||
gamma=gamma,
|
||||
train_freq=train_freq,
|
||||
gradient_steps=gradient_steps,
|
||||
n_episodes_rollout=n_episodes_rollout,
|
||||
action_noise=action_noise,
|
||||
policy_kwargs=policy_kwargs,
|
||||
tensorboard_log=tensorboard_log,
|
||||
verbose=verbose,
|
||||
device=device,
|
||||
create_eval_env=create_eval_env,
|
||||
seed=seed,
|
||||
optimize_memory_usage=optimize_memory_usage,
|
||||
# Remove all tricks from TD3 to obtain DDPG:
|
||||
# we still need to specify target_policy_noise > 0 to avoid errors
|
||||
policy_delay=1,
|
||||
target_noise_clip=0.0,
|
||||
target_policy_noise=0.1,
|
||||
_init_setup_model=False,
|
||||
)
|
||||
|
||||
# Use only one critic
|
||||
if "n_critics" not in self.policy_kwargs:
|
||||
self.policy_kwargs["n_critics"] = 1
|
||||
|
||||
if _init_setup_model:
|
||||
self._setup_model()
|
||||
|
||||
def learn(
|
||||
self,
|
||||
total_timesteps: int,
|
||||
callback: MaybeCallback = None,
|
||||
log_interval: int = 4,
|
||||
eval_env: Optional[GymEnv] = None,
|
||||
eval_freq: int = -1,
|
||||
n_eval_episodes: int = 5,
|
||||
tb_log_name: str = "DDPG",
|
||||
eval_log_path: Optional[str] = None,
|
||||
reset_num_timesteps: bool = True,
|
||||
) -> OffPolicyAlgorithm:
|
||||
|
||||
return super(DDPG, self).learn(
|
||||
total_timesteps=total_timesteps,
|
||||
callback=callback,
|
||||
log_interval=log_interval,
|
||||
eval_env=eval_env,
|
||||
eval_freq=eval_freq,
|
||||
n_eval_episodes=n_eval_episodes,
|
||||
tb_log_name=tb_log_name,
|
||||
eval_log_path=eval_log_path,
|
||||
reset_num_timesteps=reset_num_timesteps,
|
||||
)
|
||||
2
stable_baselines3/ddpg/policies.py
Normal file
2
stable_baselines3/ddpg/policies.py
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
# DDPG can be view as a special case of TD3
|
||||
from stable_baselines3.td3.policies import CnnPolicy, MlpPolicy # noqa:F401
|
||||
|
|
@ -28,10 +28,13 @@ class DQN(OffPolicyAlgorithm):
|
|||
:param batch_size: (int) Minibatch size for each gradient update
|
||||
:param tau: (float) the soft update coefficient ("Polyak update", between 0 and 1) default 1 for hard update
|
||||
:param gamma: (float) the discount factor
|
||||
:param train_freq: (int) Update the model every ``train_freq`` steps.
|
||||
:param gradient_steps: (int) How many gradient update after each step
|
||||
:param train_freq: (int) Update the model every ``train_freq`` steps. Set to `-1` to disable.
|
||||
:param gradient_steps: (int) How many gradient steps to do after each rollout
|
||||
(see ``train_freq`` and ``n_episodes_rollout``)
|
||||
Set to ``-1`` means to do as many gradient steps as steps done in the environment
|
||||
during the rollout.
|
||||
:param n_episodes_rollout: (int) Update the model every ``n_episodes_rollout`` episodes.
|
||||
Note that this cannot be used at the same time as ``train_freq``
|
||||
Note that this cannot be used at the same time as ``train_freq``. Set to `-1` to disable.
|
||||
:param optimize_memory_usage: (bool) Enable a memory efficient variant of the replay buffer
|
||||
at a cost of more complexity.
|
||||
See https://github.com/DLR-RM/stable-baselines3/issues/37#issuecomment-637501195
|
||||
|
|
|
|||
|
|
@ -35,10 +35,13 @@ class SAC(OffPolicyAlgorithm):
|
|||
:param batch_size: (int) Minibatch size for each gradient update
|
||||
:param tau: (float) the soft update coefficient ("Polyak update", between 0 and 1)
|
||||
:param gamma: (float) the discount factor
|
||||
:param train_freq: (int) Update the model every ``train_freq`` steps.
|
||||
:param gradient_steps: (int) How many gradient update after each step
|
||||
:param train_freq: (int) Update the model every ``train_freq`` steps. Set to `-1` to disable.
|
||||
:param gradient_steps: (int) How many gradient steps to do after each rollout
|
||||
(see ``train_freq`` and ``n_episodes_rollout``)
|
||||
Set to ``-1`` means to do as many gradient steps as steps done in the environment
|
||||
during the rollout.
|
||||
:param n_episodes_rollout: (int) Update the model every ``n_episodes_rollout`` episodes.
|
||||
Note that this cannot be used at the same time as ``train_freq``
|
||||
Note that this cannot be used at the same time as ``train_freq``. Set to `-1` to disable.
|
||||
:param action_noise: (ActionNoise) the action noise type (None by default), this can help
|
||||
for hard exploration problem. Cf common.noise for the different action noise type.
|
||||
:param optimize_memory_usage: (bool) Enable a memory efficient variant of the replay buffer
|
||||
|
|
@ -136,7 +139,6 @@ class SAC(OffPolicyAlgorithm):
|
|||
def _setup_model(self) -> None:
|
||||
super(SAC, self)._setup_model()
|
||||
self._create_aliases()
|
||||
assert self.critic.n_critics == 2, "SAC only supports `n_critics=2` for now"
|
||||
# Target entropy is used when learning the entropy coefficient
|
||||
if self.target_entropy == "auto":
|
||||
# automatically set target entropy if needed
|
||||
|
|
@ -218,18 +220,20 @@ class SAC(OffPolicyAlgorithm):
|
|||
with th.no_grad():
|
||||
# Select action according to policy
|
||||
next_actions, next_log_prob = self.actor.action_log_prob(replay_data.next_observations)
|
||||
# Compute the target Q value
|
||||
target_q1, target_q2 = self.critic_target(replay_data.next_observations, next_actions)
|
||||
target_q = th.min(target_q1, target_q2) - ent_coef * next_log_prob.reshape(-1, 1)
|
||||
# Compute the target Q value: min over all critics targets
|
||||
targets = th.cat(self.critic_target(replay_data.next_observations, next_actions), dim=1)
|
||||
target_q, _ = th.min(targets, dim=1, keepdim=True)
|
||||
# add entropy term
|
||||
target_q = target_q - ent_coef * next_log_prob.reshape(-1, 1)
|
||||
# td error + entropy term
|
||||
q_backup = replay_data.rewards + (1 - replay_data.dones) * self.gamma * target_q
|
||||
|
||||
# Get current Q estimates
|
||||
# Get current Q estimates for each critic network
|
||||
# using action from the replay buffer
|
||||
current_q1, current_q2 = self.critic(replay_data.observations, replay_data.actions)
|
||||
current_q_esimates = self.critic(replay_data.observations, replay_data.actions)
|
||||
|
||||
# Compute critic loss
|
||||
critic_loss = 0.5 * (F.mse_loss(current_q1, q_backup) + F.mse_loss(current_q2, q_backup))
|
||||
critic_loss = 0.5 * sum([F.mse_loss(current_q, q_backup) for current_q in current_q_esimates])
|
||||
critic_losses.append(critic_loss.item())
|
||||
|
||||
# Optimize the critic
|
||||
|
|
@ -239,8 +243,9 @@ class SAC(OffPolicyAlgorithm):
|
|||
|
||||
# Compute actor loss
|
||||
# Alternative: actor_loss = th.mean(log_prob - qf1_pi)
|
||||
qf1_pi, qf2_pi = self.critic.forward(replay_data.observations, actions_pi)
|
||||
min_qf_pi = th.min(qf1_pi, qf2_pi)
|
||||
# Mean over all critic networks
|
||||
q_values_pi = th.cat(self.critic.forward(replay_data.observations, actions_pi), dim=1)
|
||||
min_qf_pi, _ = th.min(q_values_pi, dim=1, keepdim=True)
|
||||
actor_loss = (ent_coef * log_prob - min_qf_pi).mean()
|
||||
actor_losses.append(actor_loss.item())
|
||||
|
||||
|
|
|
|||
|
|
@ -35,10 +35,13 @@ class TD3(OffPolicyAlgorithm):
|
|||
:param batch_size: (int) Minibatch size for each gradient update
|
||||
:param tau: (float) the soft update coefficient ("Polyak update", between 0 and 1)
|
||||
:param gamma: (float) the discount factor
|
||||
:param train_freq: (int) Update the model every ``train_freq`` steps.
|
||||
:param gradient_steps: (int) How many gradient update after each step
|
||||
:param train_freq: (int) Update the model every ``train_freq`` steps. Set to `-1` to disable.
|
||||
:param gradient_steps: (int) How many gradient steps to do after each rollout
|
||||
(see ``train_freq`` and ``n_episodes_rollout``)
|
||||
Set to ``-1`` means to do as many gradient steps as steps done in the environment
|
||||
during the rollout.
|
||||
:param n_episodes_rollout: (int) Update the model every ``n_episodes_rollout`` episodes.
|
||||
Note that this cannot be used at the same time as ``train_freq``
|
||||
Note that this cannot be used at the same time as ``train_freq``. Set to `-1` to disable.
|
||||
:param action_noise: (ActionNoise) the action noise type (None by default), this can help
|
||||
for hard exploration problem. Cf common.noise for the different action noise type.
|
||||
:param optimize_memory_usage: (bool) Enable a memory efficient variant of the replay buffer
|
||||
|
|
@ -144,7 +147,6 @@ class TD3(OffPolicyAlgorithm):
|
|||
def _setup_model(self) -> None:
|
||||
super(TD3, self)._setup_model()
|
||||
self._create_aliases()
|
||||
assert self.critic.n_critics == 2, "TD3 only supports `n_critics=2` for now"
|
||||
|
||||
def _create_aliases(self) -> None:
|
||||
self.actor = self.policy.actor
|
||||
|
|
@ -169,18 +171,18 @@ class TD3(OffPolicyAlgorithm):
|
|||
noise = noise.clamp(-self.target_noise_clip, self.target_noise_clip)
|
||||
next_actions = (self.actor_target(replay_data.next_observations) + noise).clamp(-1, 1)
|
||||
|
||||
# Compute the target Q value
|
||||
target_q1, target_q2 = self.critic_target(replay_data.next_observations, next_actions)
|
||||
target_q = th.min(target_q1, target_q2)
|
||||
# Compute the target Q value: min over all critics targets
|
||||
targets = th.cat(self.critic_target(replay_data.next_observations, next_actions), dim=1)
|
||||
target_q, _ = th.min(targets, dim=1, keepdim=True)
|
||||
target_q = replay_data.rewards + (1 - replay_data.dones) * self.gamma * target_q
|
||||
|
||||
# Get current Q estimates
|
||||
current_q1, current_q2 = self.critic(replay_data.observations, replay_data.actions)
|
||||
# Get current Q estimates for each critic network
|
||||
current_q_esimates = self.critic(replay_data.observations, replay_data.actions)
|
||||
|
||||
# Compute critic loss
|
||||
critic_loss = F.mse_loss(current_q1, target_q) + F.mse_loss(current_q2, target_q)
|
||||
critic_loss = sum([F.mse_loss(current_q, target_q) for current_q in current_q_esimates])
|
||||
|
||||
# Optimize the critic
|
||||
# Optimize the critics
|
||||
self.critic.optimizer.zero_grad()
|
||||
critic_loss.backward()
|
||||
self.critic.optimizer.step()
|
||||
|
|
|
|||
|
|
@ -1 +1 @@
|
|||
0.8.0a3
|
||||
0.8.0a4
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ import shutil
|
|||
import gym
|
||||
import pytest
|
||||
|
||||
from stable_baselines3 import A2C, DQN, PPO, SAC, TD3
|
||||
from stable_baselines3 import A2C, DDPG, DQN, PPO, SAC, TD3
|
||||
from stable_baselines3.common.callbacks import (
|
||||
CallbackList,
|
||||
CheckpointCallback,
|
||||
|
|
@ -14,7 +14,7 @@ from stable_baselines3.common.callbacks import (
|
|||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("model_class", [A2C, PPO, SAC, TD3, DQN])
|
||||
@pytest.mark.parametrize("model_class", [A2C, PPO, SAC, TD3, DQN, DDPG])
|
||||
def test_callbacks(tmp_path, model_class):
|
||||
log_folder = tmp_path / "logs/callbacks/"
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from stable_baselines3 import A2C, DQN, PPO, SAC, TD3
|
||||
from stable_baselines3 import A2C, DDPG, DQN, PPO, SAC, TD3
|
||||
from stable_baselines3.common.evaluation import evaluate_policy
|
||||
from stable_baselines3.common.identity_env import IdentityEnv, IdentityEnvBox, IdentityEnvMultiBinary, IdentityEnvMultiDiscrete
|
||||
from stable_baselines3.common.noise import NormalActionNoise
|
||||
|
|
@ -31,11 +31,11 @@ def test_discrete(model_class, env):
|
|||
assert np.shape(model.predict(obs)[0]) == np.shape(obs)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("model_class", [A2C, PPO, SAC, TD3])
|
||||
@pytest.mark.parametrize("model_class", [A2C, PPO, SAC, DDPG, TD3])
|
||||
def test_continuous(model_class):
|
||||
env = IdentityEnvBox(eps=0.5)
|
||||
|
||||
n_steps = {A2C: 3500, PPO: 3000, SAC: 700, TD3: 500}[model_class]
|
||||
n_steps = {A2C: 3500, PPO: 3000, SAC: 700, TD3: 500, DDPG: 500}[model_class]
|
||||
|
||||
kwargs = dict(policy_kwargs=dict(net_arch=[64, 64]), seed=0, gamma=0.95)
|
||||
if model_class in [TD3]:
|
||||
|
|
|
|||
|
|
@ -1,15 +1,19 @@
|
|||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from stable_baselines3 import A2C, DQN, PPO, SAC, TD3
|
||||
from stable_baselines3 import A2C, DDPG, DQN, PPO, SAC, TD3
|
||||
from stable_baselines3.common.noise import NormalActionNoise, OrnsteinUhlenbeckActionNoise
|
||||
|
||||
normal_action_noise = NormalActionNoise(np.zeros(1), 0.1 * np.ones(1))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("model_class", [TD3, DDPG])
|
||||
@pytest.mark.parametrize("action_noise", [normal_action_noise, OrnsteinUhlenbeckActionNoise(np.zeros(1), 0.1 * np.ones(1))])
|
||||
def test_td3(action_noise):
|
||||
model = TD3(
|
||||
def test_deterministic_pg(model_class, action_noise):
|
||||
"""
|
||||
Test for DDPG and variants (TD3).
|
||||
"""
|
||||
model = model_class(
|
||||
"MlpPolicy",
|
||||
"Pendulum-v0",
|
||||
policy_kwargs=dict(net_arch=[64, 64]),
|
||||
|
|
@ -70,6 +74,15 @@ def test_sac(ent_coef):
|
|||
model.learn(total_timesteps=1000, eval_freq=500)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("n_critics", [1, 3])
|
||||
def test_n_critics(n_critics):
|
||||
# Test SAC with different number of critics, for TD3, n_critics=1 corresponds to DDPG
|
||||
model = SAC(
|
||||
"MlpPolicy", "Pendulum-v0", policy_kwargs=dict(net_arch=[64, 64], n_critics=n_critics), learning_starts=100, verbose=1
|
||||
)
|
||||
model.learn(total_timesteps=1000)
|
||||
|
||||
|
||||
def test_dqn():
|
||||
model = DQN(
|
||||
"MlpPolicy",
|
||||
|
|
|
|||
|
|
@ -9,19 +9,13 @@ import numpy as np
|
|||
import pytest
|
||||
import torch as th
|
||||
|
||||
from stable_baselines3 import A2C, DQN, PPO, SAC, TD3
|
||||
from stable_baselines3 import A2C, DDPG, DQN, PPO, SAC, TD3
|
||||
from stable_baselines3.common.base_class import BaseAlgorithm
|
||||
from stable_baselines3.common.identity_env import FakeImageEnv, IdentityEnv, IdentityEnvBox
|
||||
from stable_baselines3.common.save_util import load_from_pkl, open_path, save_to_pkl
|
||||
from stable_baselines3.common.vec_env import DummyVecEnv
|
||||
|
||||
MODEL_LIST = [
|
||||
PPO,
|
||||
A2C,
|
||||
TD3,
|
||||
SAC,
|
||||
DQN,
|
||||
]
|
||||
MODEL_LIST = [PPO, A2C, TD3, SAC, DQN, DDPG]
|
||||
|
||||
|
||||
def select_env(model_class: BaseAlgorithm) -> gym.Env:
|
||||
|
|
|
|||
Loading…
Reference in a new issue