stable-baselines3/setup.py
Noah 96b771f24e
Implement DQN (#28)
* Created DQN template according to the paper.
Next steps:
- Create Policy
- Complete Training
- Debug

* Changed Base Class

* refactor save, to be consistence with overriding the excluded_save_params function. Do not try to exclude the parameters twice.

* Added simple DQN policy

* Finished learn and train function
- missing correct loss computation

* changed collect_rollouts to work with discrete space

* moved discrete space collect_rollouts to dqn

* basic dqn working

* deleted SDE related code

* added gradient clipping and moved greedy policy to policy

* changed policy to implement target network
and added soft update(in fact standart tau is 1 so hard update)

* fixed policy setup

* rebase target_update_intervall on _n_updates

* adapted all tests
all tests passing

* Move to stable-baseline3

* Fixes for DQN

* Fix tests + add CNNPolicy

* Allow any optimizer for DQN

* added some util functions to create a arbitrary linear schedule, fixed pickle problem with old exploration schedule

* more documentation

* changed buffer dtype

* refactor and document

* Added Sphinx Documentation
Updated changelog.rst

* removed custom collect_rollouts as it is no longer necessary

* Implemented suggestions to clean code and documentation.

* extracted some functions on tests to reduce duplicated code

* added support for exploration_fraction

* Fixed exploration_fraction

* Added documentation

* Fixed get_linear_fn -> proper progress scaling

* Merged master

* Added nature reference

* Changed default parameters to https://www.nature.com/articles/nature14236/tables/1

* Fixed n_updates to be incremented correctly

* Correct train_freq

* Doc update

* added special parameter for DQN in tests

* different fix for test_discrete

* Update docs/modules/dqn.rst

Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>

* Update docs/modules/dqn.rst

Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>

* Update docs/modules/dqn.rst

Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>

* Added RMSProp in optimizer_kwargs, as described in nature paper

* Exploration fraction is inverse of 50.000.000 (total frames) / 1.000.000 (frames with linear schedule) according to nature paper

* Changelog update for buffer dtype

* standard exlude parameters should be always excluded to assure proper saving only if intentionally included by ``include`` parameter

* slightly more iterations on test_discrete to pass the test

* added param use_rms_prop instead of mutable default argument

* forgot alpha

* using huber loss, adam and learning rate 1e-4

* account for train_freq in update_target_network

* Added memory check for both buffers

* Doc updated for buffer allocation

* Added psutil Requirement

* Adapted test_identity.py

* Fixes with new SB3 version

* Fix for tensorboard name

* Convert assert to warning and fix tests

* Refactor off-policy algorithms

* Fixes

* test: remove next_obs in replay buffer

* Update changelog

* Fix tests and use tmp_path where possible

* Fix sampling bug in buffer

* Do not store next obs on episode termination

* Fix replay buffer sampling

* Update comment

* moved epsilon from policy to model

* Update predict method

* Update atari wrappers to match SB2

* Minor edit in the buffers

* Update changelog

* Merge branch 'master' into dqn

* Update DQN to new structure

* Fix tests and remove hardcoded path

* Fix for DQN

* Disable memory efficient replay buffer by default

* Fix docstring

* Add tests for memory efficient buffer

* Update changelog

* Split collect rollout

* Move target update outside `train()` for DQN

* Update changelog

* Update linear schedule doc

* Cleanup DQN code

* Minor edit

* Update version and docker images

Co-authored-by: Antonin RAFFIN <antonin.raffin@ensta.org>
2020-06-29 11:16:54 +02:00

131 lines
4.1 KiB
Python

import os
from setuptools import setup, find_packages
with open(os.path.join('stable_baselines3', 'version.txt'), 'r') as file_handler:
__version__ = file_handler.read().strip()
long_description = """
# Stable Baselines3
Stable Baselines3 is a set of improved implementations of reinforcement learning algorithms in PyTorch. It is the next major version of [Stable Baselines](https://github.com/hill-a/stable-baselines).
These algorithms will make it easier for the research community and industry to replicate, refine, and identify new ideas, and will create good baselines to build projects on top of. We expect these tools will be used as a base around which new ideas can be added, and as a tool for comparing a new approach against existing ones. We also hope that the simplicity of these tools will allow beginners to experiment with a more advanced toolset, without being buried in implementation details.
## Links
Repository:
https://github.com/DLR-RM/stable-baselines3
Medium article:
https://medium.com/@araffin/df87c4b2fc82
Documentation:
https://stable-baselines3.readthedocs.io/en/master/
RL Baselines3 Zoo:
https://github.com/DLR-RM/rl-baselines3-zoo
## Quick example
Most of the library tries to follow a sklearn-like syntax for the Reinforcement Learning algorithms using Gym.
Here is a quick example of how to train and run PPO on a cartpole environment:
```python
import gym
from stable_baselines3 import PPO
env = gym.make('CartPole-v1')
model = PPO('MlpPolicy', env, verbose=1)
model.learn(total_timesteps=10000)
obs = env.reset()
for i in range(1000):
action, _states = model.predict(obs, deterministic=True)
obs, reward, done, info = env.step(action)
env.render()
if done:
obs = env.reset()
```
Or just train a model with a one liner if [the environment is registered in Gym](https://github.com/openai/gym/wiki/Environments) and if [the policy is registered](https://stable-baselines3.readthedocs.io/en/master/guide/custom_policy.html):
```python
from stable_baselines3 import PPO
model = PPO('MlpPolicy', 'CartPole-v1').learn(10000)
```
""" # noqa:E501
setup(name='stable_baselines3',
packages=[package for package in find_packages()
if package.startswith('stable_baselines3')],
package_data={
'stable_baselines3': ['py.typed', 'version.txt']
},
install_requires=[
'gym>=0.17',
'numpy',
'torch>=1.4.0',
# For saving models
'cloudpickle',
# For reading logs
'pandas',
# Plotting learning curves
'matplotlib'
],
extras_require={
'tests': [
# Run tests and coverage
'pytest',
'pytest-cov',
'pytest-env',
'pytest-xdist',
# Type check
'pytype',
# Lint code
'flake8>=3.8'
],
'docs': [
'sphinx',
'sphinx-autobuild',
'sphinx-rtd-theme',
# For spelling
'sphinxcontrib.spelling',
# Type hints support
# 'sphinx-autodoc-typehints'
],
'extra': [
# For render
'opencv-python',
# For atari games,
'atari_py~=0.2.0', 'pillow',
# Tensorboard support
'tensorboard',
# Checking memory taken by replay buffer
'psutil'
]
},
description='Pytorch version of Stable Baselines, implementations of reinforcement learning algorithms.',
author='Antonin Raffin',
url='https://github.com/DLR-RM/stable-baselines3',
author_email='antonin.raffin@dlr.de',
keywords="reinforcement-learning-algorithms reinforcement-learning machine-learning "
"gym openai stable baselines toolbox python data-science",
license="MIT",
long_description=long_description,
long_description_content_type='text/markdown',
version=__version__,
)
# python setup.py sdist
# python setup.py bdist_wheel
# twine upload --repository-url https://test.pypi.org/legacy/ dist/*
# twine upload dist/*