This repository focuses on exploring how Intrinsic Motivation (IM) effects Reinforcement Learning (RL) policy-based models. We use a Deep Q-Network (DQN) to simulate off-policy and Proximal Policy Optimization (PPO) as on-policy.
(Note) functionality for Rainbow Deep Q-Networks (RDQNs) is available but is not used during the study due to time constraints.
We focus on two IM methods: curiosity and empowerment, and evaluate their performance on three Atari games: Space Invaders, Q*bert, and Montezuma's Revenge. Our research aims to provide intuition on how IM methods affect these popular on-policy and off-policy agents.
Figure 1. Examples of each environment. From left to right: Space Invaders, Q*bert, and Montezuma's Revenge (Source: Gym Docs).
The file structure for the artefact is outlined below.
+-- agents
| +-- _agent.py
| +-- dqn.py
| +-- ppo.py
| +-- rainbow.py
+-- core
| +-- README.md
| +-- buffer.py
| +-- create.py
| +-- enums.py
| +-- env_details.py
| +-- exceptions.py
| +-- parameters.py
| +-- template.yaml
+-- intrinsic
| +-- _im_base.py
| +-- controller.py
| +-- empower_models.py
| +-- model.py
| +-- module.py
| +-- parameters.py
+-- models
| +-- _base.py
| +-- actor_critic.py
| +-- dueling.py
| +-- linear.py
+-- tests
| +-- test_buffer.py
| +-- test_create.py
| +-- test_dqn.py
| +-- test_env_details.py
| +-- test_helper.py
| +-- test_intrisic.py
| +-- test_logger.py
| +-- test_models.py
| +-- test_ppo.py
+-- utils
| +-- dataloader.py
| +-- helper.py
| +-- init_devices.py
| +-- logger.py
| +-- model_utils.py
| +-- plotter.py
| +-- render.py
| +-- validate.py
+-- .coverage
+-- .coveragerc
+-- LICENSE
+-- README.md
+-- main.py
+-- parameters.yaml
+-- requirements.txt
+-- demo.ipynb
\agents- contains the RL algorithm implementations\core- core functionality of the artefact\intrinsic- intrinsic motivation functionality\models- neural network models for the agents\tests- unit tests for the application\utils- utility classes and functions that provide extra functionality
This project requires a Python 3.10 environment, which can be created with the following instructions:
-
Install Visual Studio Build Tools (required for SuperSuit).
-
Create (and activate) a new environment.
- Linux or Mac
conda create --name rlatari python=3.10 source activate rlatari- Windows
conda create --name rlatari python=3.10 conda activate rlatari
-
Clone the repository, navigate to the
rl_atari_games/folder and install the required dependencies.(Note) a requirements.txt file is accessible within this folder detailing a list of the required dependencies.
git clone https://github.com/Achronus/rl_atari_games.git cd rl_atari_games conda install -c conda-forge jupyterlab conda install pytorch torchvision pytorch-cuda=11.7 -c pytorch -c nvidia pip install gymnasium[atari,accept-rom-license,other] pip install -r requirements.txt -
Create an IPython kernel for the
rlatarienvironment.python -m ipykernel install --user --name rlatari --display-name "rlatari" -
Run the
jupyter-labcommand to start JupyterLab and access the Jupyter Notebook namedrl_with_atari.ipynb, or run themain.pyfile.(Note) running the
main.pyfile will train the models. It is advised to examine this file before running it.
The algorithms created in this repository have been inspired and adapted from the following developers:
Rainbow DQN -
PPO -
Intrinsic Motivation -
- Alexander Zai and Brandon Brown: Deep Reinforcement Learning In Action, Chapter 8: Curiosity-Driven Exploration (book)
- navneet-nmk: pytorch-rl (GitHub) - for empowerment
A special thanks to each of these developers for creating incredible work in the RL space. Please check out their original work using the links above.