Docker vs Nix for the CI

[mzabaluev]

This attempts to capture the points raised in Zoom discussions about the
choice between Nix or Docker as the basis for CI infrastructure.
Currently, Docker images and docker-compose are used to set up the environment
for running integration tests against various chain node counterparts.

Description of the toolkits

Docker

Docker is an easy to use, multi-platform toolkit to organize containers.
Containers, as defined by the Open Container Intiative specifications
which Docker largely follows, provide applications with isolation from the
host system on the filesystem and process space level. The container image
format allows deployment of packaged software images in a container runtime.

• A container's file and process space are isolated from the host system.
  Any directories shared with the host or persisted as volumes
  need to be explicitly mounted.
• Builds are not generally reproducible. The primary objects of reference are
  built container images, addressed by content-based hashes.
• Dockerfiles are 🐒 simple imperative scripts for building images.
• Rich ecosystem of container images available for immediate use, or as a base
  image to build upon, or as a build stage providing tools.
• Good support on Mac (both Intel and ARM64), running Linux containers in a VM.
• Virtual networking available out of the box.
  • Simulation of networked setups not limited to localhost or the host's
    assigned IP address(es).
  • Virtual network plugins allow advanced network testing,
    e.g. simulating loss of connectivity.
• Simple declarative orchestration for development uses via docker-compose.
• Available out of the box in most cloud-hosted CI environments.
  • Image caching is typically provided too.

Nix

Nix is a package manager with a pure functional approach to building packages.

• nix-shell enables running a shell environment specified by a package (e.g.
  to bring the specific binary into the executable path).
  Otherwise, no isolation from the host system.
• Aims at fully reproducible builds with cryptographically certified provenance.
  • This can be a leaky assumption depending on the build tools in use, and may
    require attention to non-obvious inputs in the Nix flake manifest.
  • Cargo is OK except pathological cases (wacky build scripts,
    path dependencies to git submodules).
• A steeper learning curve. There is seemingly a smaller pool of people
  proficient with Nix, wide adoption is lagging.
• Ecosystem? (TODO: provide an assessment wrt. our CI needs)
• MacOS used to be problematic with Nix, but is now fully supported.
  The installer performs some necessary system-wide modifications under the hood.
• Virtual networking is not provided by Nix.
• Process orchestration is not provided by Nix.
• Requires additional actions/setup in popular cloud-hosted CI (example with GitHub Actions).
  • Unknown impact on the time of CI runs, though caching is available at least in GitHub Actions.

As a software distribution medium

Before going into the CI matter specifically, it's worth considering whether
Informal should publish its software artefacts on a Docker repository or as
a Nix project.

MZ: I think the answer should be: yes, both. We are nearly there already,
with informalsystems/hermes and cephalopodequipment/hermes on Docker Hub
and Cosmos.nix for NixOS.

Both kinds of artefacts are in demand by different infrastructure operators.
Cosmos.nix has the cool factor, and has been promoted by @soareschen as a way of
distributing software that a true blockchain company should provide to the world.

Choice for the CI solution

While the ideas behind NixOS are remarkable, is full source accountability something
that our CI testing processes require in order to justify the switch?
We might as well trust our Docker image repository to have the correct software,
provided that we use official base images to build on and our own operations are
secure.

There are other important factors to consider:

• Ease of maintenance. It would be good to avoid becoming bottlenecked on a few
  "high priests" who have the required skills.
• Extra features important for CI workflows:
  • Virtual networking to simulate real-world deployment and connectivity
    scenarios.
  • Declarative manifests vs. scripting (or an extra solution like Ansible?)
    as a way to reproducibly orchestrate the testing environment.