feat(spanner): implement generation and propagation of "x-goog-spanner-request-id" Header

[odeke-em]

This change adds sending over the "x-goog-spanner-request-id" header
for every unary and streaming call, in the form:

    <processId>.<clientId>.<requestCountForClient>.<channelId>.<rpcCountForRequest>

where:

• processId is a randomly generated uint32 singleton for the lifetime of a process
• clientId is the monotonically increasing id/number of gRPC Spanner clients created
• requestCountForClient is the monotonically increasing number of requests made by the client
• channelId currently at 1 is the Id of the client for Go
• rpcCountForRequest is the number of RPCs/retries within a specific request

This header is to be sent on both unary and streaming calls and it'll
help debug latencies for customers. After this change, the next phase shall
be providing a mechanism for customers to consume the requestID and log it
along with the documentation for how to accomplish that.

Updates #11073

[odeke-em]

Kindly cc-ing @olavloite @tharoldD @willpoint

[odeke-em]

Kindly cc-ing you @harshachinta.

[rahul2393]

@odeke-em can you please fix the vet and tests issues here?

[olavloite]

What do you mean with this? PartitionQuery retries if it receives an UNAVAILABLE error (same as most unary RPCs). See https://github.com/googleapis/googleapis/blob/master/google/spanner/v1/spanner_grpc_service_config.json for the default RPC configuration.

[olavloite]

This seems to assume that there will be only one active request for a gRPC client at the same time. That does not seem correct for two reasons:

1. For multiplexed sessions, we keep a pool of 4 (or more correct: numChannels) grpcSpannerClients. These clients are shared across all goroutines that use multiplexed sessions.
2. Regular sessions can also execute requests in parallel. Those requests would also use the same grpcSpannerClient, meaning that keeping track of for example the number of (retry) attempts at the grpcSpannerClient level won't work.

[olavloite]

This should not be fixed at 1. For regular sessions, we are setting the channel that should be used here: https://github.com/googleapis/google-cloud-go/blob/main/spanner/sessionclient.go#L404

For multiplexed sessions, we do that here:

google-cloud-go/spanner/session.go

Line 1102 in 45e1ce7

p.multiplexSessionClientCounter = p.multiplexSessionClientCounter % len(p.clientPool)

[olavloite]

The above also shows why the current strategy of assuming that a grpcSpannerClient is not used in parallel by multiple goroutines is incorrect, as the the client library just keeps a pool of numChannels (default: 4) grpcSpannerClient instances for multiplexed sessions. These will be handed out in round-robin fashion to application goroutines that want to execute a query or transaction.

[olavloite]

The raw *vkit.Client is the channel, so in that sense, this could be said to be redundant. But that property does not have a simple number or other simple string representation, which means that it is probably better/easier to just use the channel pool index that was used to fetch the channel as the channel ID here. Which again means that this property is not redundant and should be assigned a value.

[olavloite]

Same question here as for PartitionQuery; I don't quite understand what you mean with 'not automatically retried' in these cases. Could you elaborate a bit on that?

[olavloite]

This seems to confuse two different types of retries:

1. Single RPCs are retried if they receive a retryable error (e.g. UNAVAILABLE). This is handled by the gRPC libraries and transparent to the Spanner client. This attempt number should be based on the number of retries that the RPC is being retried by the gRPC libraries.
2. Read/write transactions are retried if Spanner returns an ABORTED error. That will cause the entire transaction to be retried. That should not affect the attempt number that is used for an RPC.