Add documentation about system socket

[EmilyShepherd]

No description provided.

[sftim]

What needs documenting? Is this something another contributor could pick up?

[EmilyShepherd]

What needs documenting?

Mostly that it exists 😆 I would be happy to receive contributions on it (or anything else), but it's mostly in my brain at the moment!

I'll brain dump here so you have context:

The system socket serves two purposes:

1. To make graceful node shutdown work in kiOS
2. To allow services to restart cri-o and kubelet (when appropriate)

Node Shutdown

Kubernetes' vanilla implementation of graceful node shutdown relies on systemd being present - it attempts to aquire an inhibit lock by talking over the dbus. kiOS has neither systemd, nor the dbus, so instead there is a much more basic unix socket which kubelet on kiOS is patched to use instead.

Notable differences here is that (currently) while under systemd the calling process has to request a lock for x seconds, and systemd can be configured to refuse such a request, kiOS does not care and is instead happy to trust kubelet to do things properly. As a result, kubelet's own shutdownGracePeriod configuration value can be relied upon as the shutdown grace period you'll get.

Restarts

As kiOS core's ethos is effectively just config on disk, or nothing, for most contexts cloud / environment specific bootstrapping is done via a bootstrap container. The flow being:

• kiOS starts up
• kiOS starts kubelet using whatever config is on disk, or using default values (standalone mode) in the absense of config
• One of more static pods with one or more containers perform some form of system configuration
• When ready, one of the containers reboots kubelet to use the new configuration (or cri-o, depending on what it has configured).

This could be implemented by just giving bootstrap containers access to the hostPID, but that's a bit horrible, so instead the kiOS system socket also supports a command to restart kubelet / cri-o, allowing the bootstrap container flow without risking them smashing up the rest of the system too much!

Commands and events

The system socket accepts the following commands:

• Shutdown - signal to kiOS that you want to shutdown (akin to sending a signal to process 1)
• Continue Shutdown - used by kubelet to indicate it is finished with its shutdown routine. This causes init to actually shutdown (ie kill remaining processes, unmount everything, sync, and ask the kernel to shutdown)
• Restart Kubelet - kills the kubelet and allows it to restart
• Restart cri-o - kills cri-o and allows it to restart

The system socket sends out the following events:

• Shutdown. Sent when process 1 receives a SIGTERM or if someone sends the Shutdown command over the socket. Expected to be used by kubelet.

Pitch Level

One of the biggest challenges I have with some of kiOS' documentation is that I'm not 100% sure what audience to pitch things at. There are many people who just want kubernetes clusters to "just work ™️" so are less interested in the intricacies of container runtimes / bootstrap containers etc. There are others who may be interested, but are unaware of the systemd graceful node shutdown / shutdown inhibit lock process or the role init normally plays in a system shutdown, so might not have the context to understand why / when the socket is useful.

If you have any thoughts on the tone / level to pitch at, that would be very appreciated.