From f0382a25bdbed62dcc31875d3e540ad95c1575a5 Mon Sep 17 00:00:00 2001 From: Antonin RAFFIN Date: Thu, 2 Mar 2023 14:14:57 +0100 Subject: [PATCH] Add documentation about default network architecture (#1353) * Add documentation about default network architecture * [ci skip] Rename custom policy section to Policy Networks --- .gitlab-ci.yml | 23 ----------------------- docs/guide/custom_policy.rst | 26 ++++++++++++++++++++++++-- docs/index.rst | 2 +- docs/misc/changelog.rst | 2 ++ 4 files changed, 27 insertions(+), 26 deletions(-) delete mode 100644 .gitlab-ci.yml diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml deleted file mode 100644 index ad9bfaa..0000000 --- a/.gitlab-ci.yml +++ /dev/null @@ -1,23 +0,0 @@ -image: stablebaselines/stable-baselines3-cpu:1.4.1a0 - -type-check: - script: - - pip install pytype mypy --upgrade - - make type - -pytest: - script: - - pip install tqdm rich # for progress bar - - python --version - # MKL_THREADING_LAYER=GNU to avoid MKL_THREADING_LAYER=INTEL incompatibility error - - MKL_THREADING_LAYER=GNU make pytest - coverage: '/^TOTAL.+?(\d+\%)$/' - -doc-build: - script: - - make doc - -lint-check: - script: - - make check-codestyle - - make lint diff --git a/docs/guide/custom_policy.rst b/docs/guide/custom_policy.rst index dae6048..80f6939 100644 --- a/docs/guide/custom_policy.rst +++ b/docs/guide/custom_policy.rst @@ -1,7 +1,7 @@ .. _custom_policy: -Custom Policy Network -===================== +Policy Networks +=============== Stable Baselines3 provides policy networks for images (CnnPolicies), other type of input features (MlpPolicies) and multiple different inputs (MultiInputPolicies). @@ -51,6 +51,28 @@ Each of these network have a features extractor followed by a fully-connected ne .. image:: ../_static/img/sb3_policy.png +Default Network Architecture +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The default network architecture used by SB3 depends on the algorithm and the observation space. +You can visualize the architecture by printing ``model.policy`` (see `issue #329 `_). + + +For 1D observation space, a 2 layers fully connected net is used with: + +- 64 units (per layer) for PPO/A2C/DQN +- 256 units for SAC +- [400, 300] units for TD3/DDPG (values are taken from the original TD3 paper) + +For image observation spaces, the "Nature CNN" (see code for more details) is used for feature extraction, and SAC/TD3 also keeps the same fully connected network after it. +The other algorithms only have a linear layer after the CNN. +The CNN is shared between actor and critic for A2C/PPO (on-policy algorithms) to reduce computation. +Off-policy algorithms (TD3, DDPG, SAC, ...) have separate feature extractors: one for the actor and one for the critic, since the best performance is obtained with this configuration. + +For mixed observations (dictionary observations), the two architectures from above are used, i.e., CNN for images and then two layers fully-connected network +(with a smaller output size for the CNN). + + Custom Network Architecture ^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/index.rst b/docs/index.rst index b3ec4cc..b501939 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -44,8 +44,8 @@ Main Features guide/algos guide/examples guide/vec_envs - guide/custom_env guide/custom_policy + guide/custom_env guide/callbacks guide/tensorboard guide/integrations diff --git a/docs/misc/changelog.rst b/docs/misc/changelog.rst index 9e2934e..bf2e60a 100644 --- a/docs/misc/changelog.rst +++ b/docs/misc/changelog.rst @@ -42,6 +42,7 @@ Others: - Fixed ``tests/test_vec_normalize.py`` type hint - Fixed ``stable_baselines3/common/monitor.py`` type hint - Added tests for StackedObservations +- Removed Gitlab CI file Documentation: ^^^^^^^^^^^^^^ @@ -50,6 +51,7 @@ Documentation: - Fixed typo in ``A2C`` docstring (@AlexPasqua) - Renamed timesteps to episodes for ``log_interval`` description (@theSquaredError) - Removed note about gif creation for Atari games (@harveybellini) +- Added information about default network architecture Release 1.7.0 (2023-01-10) --------------------------