Troubleshoot_Common_Vault_Cluster_Issues.md

Troubleshoot Common Vault Cluster Issues

Search for underlying issues causing unhealthy Vault clusters. Check the Vault statefulset and various pod logs to determine what is impacting the health of the Vault.

Procedure

1. (ncn-mw#) View the Vault statefulset.

   kubectl -n vault get statefulset --show-labels

   Example output:

   NAME         READY   AGE   LABELS
   cray-vault   3/3     8d    app.kubernetes.io/name=vault,vault_cr=cray-vault
   

2. (ncn-mw#) Check the pod logs for the bank-vaults container for Vault statefulset pods.

   kubectl logs -n vault cray-vault-0 --tail=-1 --prefix -c bank-vaults
   kubectl logs -n vault cray-vault-1 --tail=-1 --prefix -c bank-vaults
   kubectl logs -n vault cray-vault-0 --tail=-1 --prefix -c bank-vaults

3. (ncn-mw#) Check the Vault container logs within the pod.

   kubectl logs -n vault cray-vault-0 --tail=-1 --prefix -c vault
   kubectl logs -n vault cray-vault-1 --tail=-1 --prefix -c vault
   kubectl logs -n vault cray-vault-2 --tail=-1 --prefix -c vault

4. (ncn-mw#) Check the Vault operator pod logs using ephemerally named pods.

   kubectl logs -n vault cray-vault-operator-7dbbdbb68b-zvg2g --tail=-1 --prefix