Quentin Gallouédec
c5adad82b2
Multiprocessing support for HerReplayBuffer ( #704 )
...
* IM compat. modif from old fork
* mp her working, without offline sampling
* update readme and doc
* fix discrete action/obs space case
* handle offline sampling
* fix pos to be consistent with the old version
* improve typing and docstring
* fix discrete obs special case
* new her, using episode uid
* deal with full buffer
* offline not implemented
* info storage; compute_reward as arg; offline sampling error
* offline sampling; timeout_termination; fix last_trans detection
* rm max_episode_length from tests
* fix loading and loading test
* Fix episode sampling strategy
* Episode interrupted not valid
* Typo
* Fix infos sampling, next_obs desired goals, offline sampling
* update tests for multienvs
* speed up code
* handle timeout sampling when samping
* give up ep_uid for ep_start and ep_lenght
* speed up sampling
* Improve docstring
* Typos and renaming
* Fix typing
* Fix linter warnings
* Renaming + add note
* fix reward type
* Fix future sampling strategy
* Fix future goal selection strategy
* env_fn as lambda
* Re-fix linter warnings
* Formatting
* Fix offline sampling
* restore the initial performance budget
* Remove max_episode_length for HerReplayBuffer kwargs
* SubprcVecEnv compat test
* Dedicated SubrocVecEnv test rm n_envs from parametrization
* Back to using the env arg instead of compute_reward
* Up VecEnv import
* fix lint warnings
* fix docstring
* Fix device issue
* actor_loss_modifier in SAV and TD3
* Merge RewardModifier and ActorLossModifier into Surgeon
* update surgeon for rnd
* fix uninteded merge
* fix uninteded merge
* fix unintended merge
* Rm unintended merge
* Fix KeyError
* Remove useless `all_inds`
* Minor docstring format
* Fix hint
* speedup!
* Speedup again
* speedup
* np.nonzero
* fix env normalization
* flat sampling for speedup
* typo
* drop online
* format
* remove observation from env_cheker (see #1335 )
* update changelog
* default device to "auto"
* add comment for info storage
* add comment for ep_start and ep_length attributes
* a[b][c] to a[b, c]
* comment flatnonzero and unravel_index
* update _sample_goals docstring
* Fix future gaol sampling for split episode
* add informative error message for learning_starts too small
* use keyword arg for env
* try fix pytye
* Update stable_baselines3/common/off_policy_algorithm.py
Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>
* Add `copy_info_dict` option
* Ignore pytype
* Update changelog
* Rename variables and improve documentation
* Ignore new bug bear rule
* Add note about future strategy
* Add deprecation warning
* Fix bug trying to pickle buffer kwargs
---------
Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>
2023-03-20 12:03:57 +01:00
Antonin RAFFIN
470771b5c2
Fix Atari Roms download, enable RUF linting ( #1379 )
...
* Add extra no Atari and fix CI for forks
* Enable ruff rules
* Change to no roms
2023-03-12 18:47:52 +01:00
Quentin Gallouédec
82bc63fca4
Upgrade black formatting ( #1310 )
...
* apply black
* Reformat tests
---------
Co-authored-by: Antonin Raffin <antonin.raffin@ensta.org>
2023-02-02 11:58:41 +01:00
Sidney Tio
304c17dc78
Add append mode to Monitor ( #1037 )
...
* Added option to override or use existing CSVs
* Updated changelog for Monitor override
* Changed default value to override
* Simplify code and add test
* Update version
* Fix for pytype
Co-authored-by: Antonin Raffin <antonin.raffin@ensta.org>
2022-08-31 11:53:44 +02:00
Quentin Gallouédec
73822c34da
Support for device=auto buffers and set it as default value ( #1009 )
...
* Default device is "auto" for buffer + auto device support in BufferBaseClass
* Update docstring
* Update tests
* Unify tests
* Update changelog
* Fix tests on CUDA device
Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>
Co-authored-by: Antonin Raffin <antonin.raffin@dlr.de>
2022-08-16 17:54:55 +02:00
Ram Rachum
d64bcb401a
Fix exception cause in base_class.py ( #940 )
2022-06-21 20:58:02 +01:00
Antonin RAFFIN
a6f5049a99
Upgrade code to Python 3.7+ syntax using pyupgrade ( #887 )
...
* Upgrade code to Python 3.7+ syntax
* Update changelog
2022-04-25 13:01:38 +03:00
Quentin Gallouédec
16703b1314
Fix HER goal selection ( #848 )
...
* Goal sampled from next_achieved_goal instead of achived_goal
* No need to have special case for future anymore
* Update changelog
Co-authored-by: Antonin Raffin <antonin.raffin@ensta.org>
2022-04-11 17:50:02 +02:00
Quentin Gallouédec
db5366fb51
None as default value for env in HerReplayBuffer.sample + DQN batch size typing fix (#790 )
...
* `env` to `None` by default in `HerReplayBuffer.sample` (#788 )
* Fix DQN batch_size typing
* Fix changelog
Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>
Co-authored-by: Antonin Raffin <antonin.raffin@dlr.de>
2022-02-24 15:51:01 +01:00
Jaden Travnik
75b6f3b3b0
Dictionary Observations ( #243 )
...
* First commit
* Fixing missing refs from a quick merge from master
* Reformat
* Adding DictBuffers
* Reformat
* Minor reformat
* added slow dict test. Added SACMultiInputPolicy for future. Added private static image transpose helper to common policy
* Ran black on buffers
* Ran isort
* Adding StackedObservations classes used within VecStackEnvs wrappers. Made test_dict_env shorter and removed slow
* Running isort :facepalm
* Fixed typing issues
* Adding docstrings and typing. Using util for moving data to device.
* Fixed trailing commas
* Fix types
* Minor edits
* Avoid duplicating code
* Fix calls to parents
* Adding assert to buffers. Updating changelong
* Running format on buffers
* Adding multi-input policies to dqn,td3,a2c. Fixing warnings. Fixed bug with DictReplayBuffer as Replay buffers use only 1 env
* Fixing warnings, splitting is_vectorized_observation into multiple functions based on space type
* Created envs folder in common. Updated imports. Moved stacked_obs to vec_env folder
* Moved envs to envs directory. Moved stacked obs to vec_envs. Started update on documentation
* Fixes
* Running code style
* Update docstrings on torch_layers
* Decapitalize non-constant variables
* Using NatureCNN architecture in combined extractor. Increasing img size in multi input env. Adding memory reduction in test
* Update doc
* Update doc
* Fix format
* Removing NineRoom env. Using nested preprocess. Removing mutable default args
* running code style
* Passing channel check through to stacked dict observations.
* Running black
* Adding channel control to SimpleMultiObsEnv. Passing check_channels to CombinedExtractor
* Remove optimize memory for dict buffers
* Update doc
* Move identity env
* Minor edits + bump version
* Update doc
* Fix doc build
* Bug fixes + add support for more type of dict env
* Fixes + add multi env test
* Add support for vectranspose
* Fix stacked obs for dict and add tests
* Add check for nested spaces. Fix dict-subprocvecenv test
* Fix (single) pytype error
* Simplify CombinedExtractor
* Fix tests
* Fix check
* Merge branch 'master' into feat/dict_observations
* Fix for net_arch with dict and vector obs
* Fixes
* Add consistency test
* Update env checker
* Add some docs on dict obs
* Update default CNN feature vector size
* Refactor HER (#351 )
* Start refactoring HER
* Fixes
* Additional fixes
* Faster tests
* WIP: HER as a custom replay buffer
* New replay only version (working with DQN)
* Add support for all off-policy algorithms
* Fix saving/loading
* Remove ObsDictWrapper and add VecNormalize tests with dict
* Stable-Baselines3 v1.0 (#354 )
* Bump version and update doc
* Fix name
* Apply suggestions from code review
Co-authored-by: Adam Gleave <adam@gleave.me>
* Update docs/index.rst
Co-authored-by: Adam Gleave <adam@gleave.me>
* Update wording for RL zoo
Co-authored-by: Adam Gleave <adam@gleave.me>
* Add gym-pybullet-drones project (#358 )
* Update projects.rst
Added gym-pybullet-drones
* Update projects.rst
Longer title underline
* Update changelog
Co-authored-by: Antonin Raffin <antonin.raffin@ensta.org>
* Include SuperSuit in projects (#359 )
* include supersuit
* longer title underline
* Update changelog.rst
* Fix default arguments + add bugbear (#363 )
* Fix potential bug + add bug bear
* Remove unused variables
* Minor: version bump
* Add code of conduct + update doc (#373 )
* Add code of conduct
* Fix DQN doc example
* Update doc (channel-last/first)
* Apply suggestions from code review
Co-authored-by: Anssi <kaneran21@hotmail.com>
* Apply suggestions from code review
Co-authored-by: Adam Gleave <adam@gleave.me>
Co-authored-by: Anssi <kaneran21@hotmail.com>
Co-authored-by: Adam Gleave <adam@gleave.me>
* Make installation command compatible with ZSH (#376 )
* Add quotes
* Add Zsh bracket info
* Add clarify pip installation line
* Make note bold
* Add Zsh pip installation note
* Add handle timeouts param
* Fixes
* Fixes (buffer size, extend test)
* Fix `max_episode_length` redefinition
* Fix potential issue
* Add some docs on dict obs
* Fix performance bug
* Fix slowdown
* Add package to install (#378 )
* Add package to install
* Update docs packages installation command
Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>
* Fix backward compat + add test
* Fix VecEnv detection
* Update doc
* Fix vec env check
* Support for `VecMonitor` for gym3-style environments (#311 )
* add vectorized monitor
* auto format of the code
* add documentation and VecExtractDictObs
* refactor and add test cases
* add test cases and format
* avoid circular import and fix doc
* fix type
* fix type
* oops
* Update stable_baselines3/common/monitor.py
Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>
* Update stable_baselines3/common/monitor.py
Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>
* add test cases
* update changelog
* fix mutable argument
* quick fix
* Apply suggestions from code review
* fix terminal observation for gym3 envs
* delete comment
* Update doc and bump version
* Add warning when already using `Monitor` wrapper
* Update vecmonitor tests
* Fixes
Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>
* Reformat
* Fixed loading of ``ent_coef`` for ``SAC`` and ``TQC``, it was not optimized anymore (#392 )
* Fix ent coef loading bug
* Add test
* Add comment
* Reuse save path
* Add test for GAE + rename `RolloutBuffer.dones` for clarification (#375 )
* Fix return computation + add test for GAE
* Rename `last_dones` to `episode_starts` for clarification
* Revert advantage
* Cleanup test
* Rename variable
* Clarify return computation
* Clarify docs
* Add multi-episode rollout test
* Reformat
Co-authored-by: Anssi "Miffyli" Kanervisto <kaneran21@hotmail.com>
* Fixed saving of `A2C` and `PPO` policy when using gSDE (#401 )
* Improve doc and replay buffer loading
* Add support for images
* Fix doc
* Update Procgen doc
* Update changelog
* Update docstrings
Co-authored-by: Adam Gleave <adam@gleave.me>
Co-authored-by: Jacopo Panerati <jacopo.panerati@utoronto.ca>
Co-authored-by: Justin Terry <justinkterry@gmail.com>
Co-authored-by: Anssi <kaneran21@hotmail.com>
Co-authored-by: Tom Dörr <tomdoerr96@gmail.com>
Co-authored-by: Tom Dörr <tom.doerr@tum.de>
Co-authored-by: Costa Huang <costa.huang@outlook.com>
* Update doc and minor fixes
* Update doc
* Added note about MultiInputPolicy in error of NatureCNN
* Merge branch 'master' into feat/dict_observations
* Address comments
* Naming clarifications
* Actually saving the file would be nice
* Fix edge case when doing online sampling with HER
* Cleanup
* Add sanity check
Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>
Co-authored-by: Anssi "Miffyli" Kanervisto <kaneran21@hotmail.com>
Co-authored-by: Adam Gleave <adam@gleave.me>
Co-authored-by: Jacopo Panerati <jacopo.panerati@utoronto.ca>
Co-authored-by: Justin Terry <justinkterry@gmail.com>
Co-authored-by: Tom Dörr <tomdoerr96@gmail.com>
Co-authored-by: Tom Dörr <tom.doerr@tum.de>
Co-authored-by: Costa Huang <costa.huang@outlook.com>
2021-05-11 12:29:30 +02:00
Megan Klaiber
852961139e
Fix bug with full HerReplayBuffer ( #236 )
...
* Fix bug with full replay buffer
* Updated changelog
* Update tests/test_her.py
Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>
2020-11-20 13:23:03 +01:00
Megan Klaiber
dd6e361204
Implement HER ( #120 )
...
* Added working her version, Online sampling is missing.
* Updated test_her.
* Added first version of online her sampling. Still problems with tensor dimensions.
* Reformat
* Fixed tests
* Added some comments.
* Updated changelog.
* Add missing init file
* Fixed some small bugs.
* Reduced arguments for HER, small changes.
* Added getattr. Fixed bug for online sampling.
* Updated save/load funtions. Small changes.
* Added her to init.
* Updated save method.
* Updated her ratio.
* Move obs_wrapper
* Added DQN test.
* Fix potential bug
* Offline and online her share same sample_goal function.
* Changed lists into arrays.
* Updated her test.
* Fix online sampling
* Fixed action bug. Updated time limit for episodes.
* Updated convert_dict method to take keys as arguments.
* Renamed obs dict wrapper.
* Seed bit flipping env
* Remove get_episode_dict
* Add fast online sampling version
* Added documentation.
* Vectorized reward computation
* Vectorized goal sampling
* Update time limit for episodes in online her sampling.
* Fix max episode length inference
* Bug fix for Fetch envs
* Fix for HER + gSDE
* Reformat (new black version)
* Added info dict to compute new reward. Check her_replay_buffer again.
* Fix info buffer
* Updated done flag.
* Fixes for gSDE
* Offline her version uses now HerReplayBuffer as episode storage.
* Fix num_timesteps computation
* Fix get torch params
* Vectorized version for offline sampling.
* Modified offline her sampling to use sample method of her_replay_buffer
* Updated HER tests.
* Updated documentation
* Cleanup docstrings
* Updated to review comments
* Fix pytype
* Update according to review comments.
* Removed random goal strategy. Updated sample transitions.
* Updated migration. Removed time signal removal.
* Update doc
* Fix potential load issue
* Add VecNormalize support for dict obs
* Updated saving/loading replay buffer for HER.
* Fix test memory usage
* Fixed save/load replay buffer.
* Fixed save/load replay buffer
* Fixed transition index after loading replay buffer in online sampling
* Better error handling
* Add tests for get_time_limit
* More tests for VecNormalize with dict obs
* Update doc
* Improve HER description
* Add test for sde support
* Add comments
* Add comments
* Remove check that was always valid
* Fix for terminal observation
* Updated buffer size in offline version and reset of HER buffer
* Reformat
* Update doc
* Remove np.empty + add doc
* Fix loading
* Updated loading replay buffer
* Separate online and offline sampling + bug fixes
* Update tensorboard log name
* Version bump
* Bug fix for special case
Co-authored-by: Antonin Raffin <antonin.raffin@dlr.de>
Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>
2020-10-22 11:56:43 +02:00
