fix(f3): try to make f3 participation stick to a single node

[Stebalien]

Related Issues

Proposed Changes

Try to make f3 participation stick to a single node. Because apparently this is a feature none of us knew about.

Checklist

Before you mark the PR ready for review, please make sure that:

• Commits have a clear commit message.
• PR title conforms with contribution conventions
• Update CHANGELOG.md or signal that this change does not need it per contribution conventions
• New features have usage guidelines and / or documentation updates in
  • Lotus Documentation
  • Discussion Tutorials
• Tests exist for new functionality or change in behavior
• CI is green

[Stebalien]

Requires #12665 to actually work in practice. That PR also has a test (not for F3, but a test that ensures that this feature works correctly).

[Stebalien]

Note: this will try to stick to a single node, but it'll gracefully move on to a new node the first time a request against the current one fails.

[masih]

But participation should work without this feature. Right? What does this give us that we need / don't have in the current participation mechanism?

[Stebalien]

Hm. I guess it'll just make it slightly more efficient? It's not actually critical.

The built-in multi-node dispatch currently works by trying all nodes in-order till once succeeds. So, e.g., node 1 returns an error when requesting a ticket, we'll get the ticket from node 2. Then we'll try to participate with node 1 (and fail), switching to node 2 (and succeeding). Every subsequent request will take two requests due to the overlapping leases.

With this change, we'll permanently switch to node 2 until node 2 returns an error. At that point, we'll walk through the available nodes until one doesn't return an error or until we've tried 5 nodes.