Hashicorp has recently shipped v1.4 of its Vault secrets management and ID management tool, which comes with lots of new and exciting features, but the one that we will focus in this blog is integrated storage. The previous Vault release integrated storage was in beta, but now with with v1.4 it is in general availability. Although Vault supports many storage backends, such as Consul, MySQL, PostgreSQL, Google Cloud Storage etc., the internal storage option eliminates the need to manage a separate storage backend. It also provide high availability, supports Enterprise Replication and provides backup/restore workflows. This blog will walk you through best practices for the new Vault deployment with integrated storage using Helm templates.
Vault’s Raft Protcol
Vault’s integrated storage uses a consensus protocol based on Raft to provide consistency. There’s a very nice visual explaination of Raft available here. To briefly explain, Raft nodes are always in one of three states: follower, candidate, or leader. All nodes initially start out as a follower. In this state, nodes can accept log entries from a leader and can cast votes. If no entries are received for some time, nodes self-promote to the candidate state. In the candidate state, nodes request votes from their peers. If a candidate receives a quorum of votes, then it is promoted to a leader. The leader must accept new log entries and replicate to all the other followers.
When Vault is started, a single server is initialized which makes the cluster size 1 so the server self-promotes and becomes leader. Then as new nodes are added to the Vault cluster, the process involves an encrypted challenge/answer workflow. To accomplish this, all nodes in a single raft cluster must share the same seal configuration. If you are using Auto Unseal then the join process can use the configured seal configuration to automatically decrypt the challenge and can provide the answer. In case of Shamir seal the unseal keys must be provided to the nodes before it can decrypt the challenge and respond with the answer.
Deploying Vault on GKE
Now we are going to deploy Vault on Google Kubernetes Engine (GKE) using the Vault Helm chart available here. We are going to use Terraform to deploy Vault on GKE. The complete terraform code is available here. This code will create a GKE cluster, generates self-signed TLS certificates for Vault, and will deploy Vault using Terraform Helm provider on the GKE cluster.
First, clone the Terraform code in any location and then also clone the Vault helm chart.
$ git clone https://github.com/mohsinrz/vault-gke-raft $ cd vault-gke-raft $ git clone https://github.com/hashicorp/vault-helm
Setup gcloud authentication for your account, create a service account that has access to use resources in your project and store the JSON for this account in the
creds folder. After that modify the
terraform.tfvars accordingly and deploy the stack.
$ terraform init $ terraform apply
After completion Terraform code will output the URL for accessing the newly deployed Vault. We can check the status of the pods using
kubectl but for that first get the credentials for the GKE cluster.
$ gcloud container clusters get-credentials demo-cluster Fetching cluster endpoint and auth data. kubeconfig entry generated for demo-cluster. $ $ kubectl get pods -n vault NAME READY STATUS RESTARTS AGE vault-0 0/1 Running 0 24m vault-1 0/1 Running 0 24m vault-2 0/1 Running 0 24m vault-agent-injector-7d4cccc866-7qfkx 1/1 Running 0 24m
The pods will not become ready until they are bootstrapped and unsealed which will involve making one of the pods as Raft leader and joining others to this pod. First we will make
vault-0 as the leader by running the following commands to initialize Vault and unseal it.
$ kubectl exec -ti vault-0 -n vault -- vault operator init
We will use the unseal keys from the output of above command to unseal Vault.
$ kubectl exec -ti vault-0 -n vault -- vault operator unseal
After the Vault is unsealed the pod will become ready and will be elected as leader (you can verify this by checking logs
kubectl logs vault-0 -n vault).
$ kubectl exec -ti vault-0 -n vault -- vault status Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 1.4.0 Cluster Name vault-cluster-5d640d05 Cluster ID 4d9f4351-52ac-4819-b0a0-ce9e2949aa87 HA Enabled true HA Cluster https://vault-0.vault-internal:8201 HA Mode active $ $ kubectl get pods -n vault NAME READY STATUS RESTARTS AGE vault-0 1/1 Running 0 2m18s vault-1 0/1 Running 0 2m18s vault-2 0/1 Running 0 2m18s vault-agent-injector-7d4cccc866-fbmz4 1/1 Running 0 2m19s
Now we will make other pods join the cluster leader
Vault-0 so they can become part of the Raft cluster before we unseal them. This step will be done for all the remaining pods.
$ kubectl exec -ti vault-1 -n vault -- vault operator raft join -leader-ca-cert="$(cat ./tls/ca-certificate.cert)" --address "https://vault-1.vault-internal:8200" "https://vault-0.vault-internal:8200" Key Value --- ----- Joined true # Unseal vault-1 kubectl exec -ti vault-1 -n vault -- vault operator unseal
$ kubectl get pods -n vault NAME READY STATUS RESTARTS AGE vault-0 1/1 Running 0 12m vault-1 1/1 Running 0 12m vault-2 1/1 Running 0 12m vault-agent-injector-7d4cccc866-5vs9w 1/1 Running 0 12m
After all the pods become part of the Raft cluster and unsealed, all pods will become ready and the Vault cluster will be ready to serve requests.
Integrated storage in HashiCorp Vault 1.4 will simplify the deployment and operation of production Vault clusters significantly. The Vault Helm chart greatly reduces the complexity of running Vault on Kubernetes, providing a repeatable deployment process. If you are already using Vault with Consul as a storage backend and want to migrate to integrated storage then you can also take a look at this guide provided by Hashicorp.
//take the first step, and reach out to the ArctiqTeam if you would like our support and guidance on your Vault projects.