Magical Deterministic Simulator for distributed systems.
MadSim is a Rust async runtime similar to tokio, but with a key feature called deterministic simulation.
The main idea is borrowed from FoundationDB and sled simulation guide. Your code should be able to deterministically executed on top of a simulator. The simulator will amplify randomness, create chaos and inject failures into your system. A lot of hidden bugs may be revealed, which you can then deterministically reproduce them until they are fixed. If your system can survive such chaos, you will have more confidence in deploying your system in the real world.
However, implementing deterministic simulation is difficult. All I/O-related interfaces must be mocked during the simulation, and all uncertainties should be eliminated. This project is created to make that easy.
A part of the implementation of this crate is inspired by tokio-rs/simulation.
See also the blog posts for a detailed writeup:
- Deterministic Simulation: A New Era of Distributed System Testing (Part 1 of 2)
- Applying Deterministic Simulation: The RisingWave Story (Part 2 of 2)
Add the following lines to your Cargo.toml:
[dependencies]
madsim = "0.2"If your project depends on the following crates, replace them by our simulators:
[dependencies]
tokio = { version = "0.2", package = "madsim-tokio" }
tonic = { version = "0.5", package = "madsim-tonic" }
etcd-client = { version = "0.4", package = "madsim-etcd-client" }
rdkafka = { version = "0.3", package = "madsim-rdkafka" }
aws-sdk-s3 = { version = "0.5", package = "madsim-aws-sdk-s3" }
[dev-dependencies]
tonic-build = { version = "0.5", package = "madsim-tonic-build" }If your dependency graph includes the following crates, replace them by our patched version:
[patch.crates-io]
quanta = { git = "https://github.com/madsim-rs/quanta.git", rev = "948bdc3" }
getrandom = { git = "https://github.com/madsim-rs/getrandom.git", rev = "8daf97e" }
tokio-retry = { git = "https://github.com/madsim-rs/rust-tokio-retry.git", rev = "95e2fd3" }
tokio-postgres = { git = "https://github.com/madsim-rs/rust-postgres.git", rev = "4538cd6" }
tokio-stream = { git = "https://github.com/madsim-rs/tokio.git", rev = "ab251ad" }When built normally, these crates are identical to the original ones.
To run your code on the simulator, enable the config madsim:
RUSTFLAGS="--cfg madsim" cargo testNow you have gotten rid of tokio/tonic and you are in the simulation world!
We provide a set of APIs to control the simulator. You can use them to kill a process, disconnect the network, inject failures, etc.
Check out the documentation and search for the madsim feature to learn more usages.
- MadRaft: The labs of Raft consensus algorithm derived from MIT 6.824 and PingCAP Talent Plan.
- RisingWave: A distributed SQL database for stream processing that uses MadSim for deterministic testing.
Apache License 2.0