Queue kernel input events

[michaelfig]

@warner and I discovered that there were reentrancy problems with the ag-solo inbound command processing. The general solution to reentrancy is to create a SwingSet input queue, as is hinted at in #710

We may want to adopt a queuing mechanism as part of SwingSet so that other hosts don't have the same problems by construction.

[warner]

@dckc and I found another need for this today: the xsnap vat manager spawns a child process and controls it with netstrings over a pipe. The initial message ("evaluate SES, load this bundle, tell me when you're ready") is kicked off by the creation of a dynamic vat. The kernel does not wait for this to complete: it assumes it will take a while and continues on to the next crank. At some point in the future, the child finishes its preparations and sends back the "I'm ready" message, causing the VatManager to resolve a Promise which results in the run-queue getting a new message (aimed at the vatAdmin vat).

If this "I'm ready" input occurs in the middle of some existing crank, things could get confused: the run-queue message will be part of the same state delta as the existing crank. If that crank were rolled back for whatever reason, the "dynamic vat is ready" message would be discarded as well. This becomes more likely to happen when all vats are running on xsnap, because they'll all be waiting for IO and thus receptive to IO for unrelated vats/workers. The consequences are not dire so far, because cranks aren't usually abandoned, but we certainly need to fix it.

The solution will be to route this "worker process is ready" event through the same input queue that we want for ag-solo inbound command processing. We don't need to do this for the other IO we do to a vat worker (the ones that start and end a crank, and the ones which process syscalls during a crank) because the kernel knows it is in the middle of a crank when those happen, and the unrelated workers should not be writing anything to the kernel at that time. (Although.. now that I think about it, if a worker process spontaneously dies, outside the context of a crank, we need to decide how and when to handle that: do we hide the fact until someone tries to talk to the vat again? do we pretend that it was terminated intentionally? do we try to re-load the vat from the previous state and mask the surprising exit?).

The other reason we want the kernel to have an input queue is for device inputs. One example is the "mailbox device", whose external API surface includes a function which the host can call when data arrives from a remote machine (perhaps in the POST handler of an HTTP server). The device reacts to this arrival by adding a message to the run-queue, which means changing kernel state, which runs all the risks described above for vat worker startup. Currently, cosmic-swingset has an external queue for this purpose, but we should move it into the kernel proper.

This overlaps with device setup (#1351) because we want the external API surface to be routed through the kernel's input queue. That would be easier if we knew what that surface was. I'm thinking the device construction function should return an object which properties are API functions, but the controller wraps each function in the input queue interlock before revealing the wrappers to the caller. Then e.g. the webserver could call mailboxAPI.addMessage(data) at any moment, without regard to the state the kernel is in, and the kernel input queue would take responsibility for delaying its processing until the kernel is between cranks.

That "input handling window" should probably be immediately after each crank is complete, after the state is committed. The window should have its own DB transaction (flush the crankBuffer, but not the blockBuffer, since the IO call still happens within the context of a block, if the host application uses blocks), and should be flushed afterwards to make sure anything added to the run-queue by the input function actually sticks. I think it's safe to drain the entire input queue after each crank, and not worry about scheduling them any slower.