Python Monorepo

A Python monorepo is a single repository containing multiple projects, making it easier to manage and update them together.

This repository is a lightweight template showcasing how to structure a Python monorepo in a scalable way.

Project structure

The root directory of a generic Python project baz in the monorepo has the following structure:

baz/
├─── src/
│    └─── org/path/to/baz
│         ├─── __init__.py
│         └─── ...
├─── tests/
│    ├─── __init__.py
│    └─── ...
├─── pyproject.toml
├─── poetry.lock
├─── Makefile
├─── .black.toml
└─── .ruff.toml

The src directory contains the source code of the project. The presence of the chain of directories org/path/to/baz reflects in the module path of the project org.path.to.baz. This effectively embeds the project as part of a single hierarchy of modules rooted in org for the user, while preserving separations from other projects in the monorepo for the maintainer.

The tests directory contains tests for the project, written using the pytest framework.

The pyproject.toml file is used by Poetry. It contains metadata for the project and its dependencies. The poetry.lock file is the resolution of dependencies generated by Poetry.

The Makefile file defines commands for the project like installation and code linting. We use Black for formatting (config file .black.toml), mypy for typing, and Ruff for other types of linting like import sorting (config file .ruff.toml).

Installing other monorepo packages as dependencies

Some of the packages in the monorepo, such as internal libraries, will be installed by other projects. To keep things simple, packages in the monorepo are not published to a package index and are instead installed by specifying their relative path in the monorepo. This makes it easy to have a clear view of the whole code we are running and promotes alignment of dependencies across different projects.

We also install monorepo packages in editable mode¹, meaning that the Python files installed in the environment are exactly those in the monorepo and not a copy of them. This makes it easy to edit libraries on the go, test them, and commit changes to git.

Note that, by installing a package with its path, we are effectively forcing all installing projects to share a single version of it: the one materialized in the last git commit. It is important to keep this in mind, as breaking changes should ideally be addressed in all installing projects in the same pull request to not break anything.

Since only the last version of a package can be installed, a CHANGELOG.md file listing changes of each version loses some of its effectiveness. Still, it might be beneficial to have one to make it easy for other devs to learn what's new in a library, especially in the case of a large monorepo.

Project assets

It might be necessary for a Python library or application to have access to some asset files to work correctly. By convention, we decide to store these assets in an assets directory located at the root of the project. This has the advantage of increased cohesiveness compared to using a location external to the project directory. It also avoids polluting the src directory with large, non-code files.

Important to note: when installing a package that uses these assets, we need to ensure installation in editable mode. This is crucial because a non-editable installation would separate the source directory from the project, hindering access to the assets. Fortunately, this aligns with our best practice of installing all monorepo packages in editable mode.

Using Breadcrumbs

There might be some cases where we would like to reference a file that requires navigating upwards in the directory structure of the monorepo.

As an example, consider the need to symlink the common configuration file .ruff.toml found at the root of the monorepo for each project root. The common solution would be to use a symlink of the form ../../.ruff.toml to point to this file. Unfortunately, having sequences of ../ in a path has a series of disadvantages. For example, after refactoring the directory structure, the path might break in unexpected and unsafe ways, potentially pointing outside of the monorepo.

To address this, we can use Breadcrumbs to turn the series of ../ into a safer, more readable .root.bc. We can then symlink the configuration file as .root.bc/.ruff.toml.

Other instances of upwards navigation in the monorepo, where Breadcrumbs might be useful, are installation of other monorepo packages via relative path and access to assets stored in the project root from the code in src.

Monorepo structure

As an example, the monorepo in this repository divides projects in two categories:

• apps: applications for production features
• libs: libraries to be included as dependency of applications or other libraries

Each monorepo should be structured according to the nature of the projects and the needs of the team.

Footnotes

1. This is accomplished by specifying develop = true when defining the Poetry dependency in pyproject.toml ↩