cuberl

cuberl is a C++ library containing implementations of various reinforcement learning, filtering and planning algorithms. The following is an indicative list of examples.

Examples

Introductory

• CubeRl Basics
• Using PyTorch C++ API Part 1
• Using PyTorch C++ API Part 2
• Using PyTorch C++ API Part 3
• Using TensorboardServer class
• Toy Markov chain

Reinforcement learning

• DummyAgent on MountainCar-v0
• Multi-armed bandit with epsilon-greedy policy
• Multi-armed bandit with soft-max policy
• Advertisement placement
• Iterative policy evaluation on FrozenLake-v0
• Policy iteration on FrozenLake-v0
• Value iteration on FrozenLake-v0
• SARSA on CliffWalking-v0
• Q-learning on CliffWalking-v0
• A2C on CartPole-v1
• DQN on Gridworld
• DQN on Gridworld with experience replay
• REINFORCE algorithm on CartPole-v0
• Expected SARSA on CliffWalking-v0
• Approximate Monte Carlo on MountainCar-v0
• Monte Carlo tree search on Taxi-v3
• Double Q-learning on CartPole-v0
• First visit Monte Carlo on FrozenLake-v0

Path planning

• Path planning with rapidly-exploring random trees (TODO)
• Path planning with dynamic windows (TODO)
• A* search on a road network from Open Street Map data

Installation

The cubeai library has a host of dependencies:

• A compiler that supports C++20 e.g. g++-11
• Boost C++
• CMake >= 3.6
• Eigen
• Gtest (if configured with tests)

In addition, the library also incorporates, see (include/cubeai/extern), the following libraries (you don't need to install these):

• HTTPRequest
• better-enums
• nlohmann/json

Enabling PyTorch and CUDA

cuberl can be complied with CUDA and/or PyTorch support. If PyTorch has been compiled using CUDA support, then you need to enable CUDA as well. In order to do so set the flag USE_CUDA in the CMakeLists.txt to ON. cuberl assumes that PyTorch is compiled with the C++11 ABI.

Documentation dependencies

There are extra dependencies if you want to generate the documentation. Namely,

• Doxygen
• Sphinx
• sphinx_rtd_theme
• breathe
• m2r2

Note that if Doxygen is not found on your system CMake will skip this. On a Ubuntu/Debian based machine, you can install Doxygen using

sudo apt-get install doxygen

Similarly, install sphinx_rtd_theme using

pip install sphinx_rtd_theme

Install breathe using

pip install breathe

Install m2r2 using

pip install m2r2

Issues

undefined reference to cudaLaunchKernelExC@libcudart.so.11.0.

You may want to check with nvidia-msi your CUDA Version and make sure it is compatible with the PyTorch library you are linking against

TypeError: Descriptors cannot be created directly.

This issue may be occur when using the TensorBoardServer in cuberl. This issue is related with an issue with protobuf. See: https://stackoverflow.com/questions/72441758/typeerror-descriptors-cannot-not-be-created-directly for possible solutions.