Skip to main content

Integrating Hashicorp Vault with Azure AKS using Azure POD Identity

Integrating Hashicorp Vault with Azure AKS using Azure POD Identity

There are a couple of challenges when we use Vault in the context of Azure Kubernetes Service or AKS. The foremost is no one wants to use static identities. That’s kind of a problem that Vaults solves by generating dynamic tokens or identities that have a lease attached to them. When the lease ends, you throw them away, so basically, you don’t store your secret data statically. Secondly, when you’re deploying resources on Azure they also shouldn’t have statically defined credentials instead, resources should utilize Managed Identities, which eliminates the need to manage credentials and allow you to authenticate to any Azure service that supports Azure AD authentication. The solution to these challenges is Azure AD Pod Identity, which allows you to execute Pods in the security context of an Azure Managed Identity. The identity is dynamically assigned to any pod that is matching certain requirements. Since Vault also supports Azure AD authentication, we will show you how we can use Azure AD Pod identity to integrate Vault with AKS.

Creating AKS cluster

Okay, time to get some work done. Let’s first create a resource group for the AKS cluster and then deploy a cluster in AKS.

$ az group create --location canadacentral --name aks-demo-rg
$ az aks create --resource-group aks-demo-rg --name demo-cluster --node-count 1 --kubernetes-version 1.17.13

To manage the AKS cluster locally, a kubeconfig, the Kubernetes configuration of the AKS cluster, will need to be cloned to a local machine.

Run the command below to copy the kubeconfig to a local computer:

$ az aks get-credentials --name demo-cluster --resource-group az-demo-rg

Enabling Azure AD Pod Identitiy on AKS

Now deploy AAD Pod Identity components on the cluster using Helm.

$ helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
$ helm install aad-pod-identity aad-pod-identity/aad-pod-identity

Once the deployment is finished, you’ll end up having two new resources deployed on your cluster.

  • Managed Identity Controller (MIC)
  • Node Managed Identity (NMI)

MIC is responsible for binding Azure Identities to pods. The NMI will act like an interceptor that observes incoming requests for your pods and will call back into Azure to acquire an access token from Azure AD to communicate with Azure APIs on behalf of that Azure Identity.

Creating Azure identity

We will create two identities, one for Vault pods and the other for a demo pod. First, we will set some environment variables which will help us in the execution of the next set of commands.

$ export SUBSCRIPTION_ID="XXXXXXXXXX"
$ export RESOURCE_GROUP="aks-demo-rg"
$ export CLUSTER_NAME="demo-cluster"
$ export CLUSTER_LOCATION="canadacluster"

$ export IDENTITY_RESOURCE_GROUP="MC_${RESOURCE_GROUP}_${CLUSTER_NAME}_${CLUSTER_LOCATION}"

Let’s create the first identity for the demo pod.

$ export IDENTITY_NAME="demo"
$ az identity create -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME}
$ export IDENTITY_CLIENT_ID="$(az identity show -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME} --query clientId -otsv)"
$ export IDENTITY_RESOURCE_ID="$(az identity show -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME} --query id -otsv)"

Now assign the role “Reader” to the identity so that it has read access to the resource group, which is required because MIC will try to acquire the access token for the identity. At the same time, store the identity assignment ID as an environment variable.

$ export IDENTITY_ASSIGNMENT_ID="$(az role assignment create --role Reader --assignee ${IDENTITY_CLIENT_ID} --scope /subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${IDENTITY_RESOURCE_GROUP} --query id -otsv)"

Repeat the above steps for the second identity for the Vault pods.

$ export IDENTITY_NAME="vault"
$ az identity create -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME}
$ export IDENTITY_CLIENT_ID="$(az identity show -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME} --query clientId -otsv)"
$ export IDENTITY_RESOURCE_ID="$(az identity show -g ${IDENTITY_RESOURCE_GROUP} -n ${IDENTITY_NAME} --query id -otsv)"
$ export IDENTITY_ASSIGNMENT_ID="$(az role assignment create --role Reader --assignee ${IDENTITY_CLIENT_ID} --scope /subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${IDENTITY_RESOURCE_GROUP} --query id -otsv)"

Deploying Azure Identity on AKS

Next, create Azure Identity in our cluster that references the identities we created above.

$ export IDENTITY_NAME="demo"
$ cat <<EOF | kubectl apply -f -
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
  name: ${IDENTITY_NAME}
spec:
  type: 0
  resourceID: ${IDENTITY_RESOURCE_ID}
  clientID: ${IDENTITY_CLIENT_ID}
EOF

In the above yaml, spec.type is set to 0, which represents a Managed Service Identity (MSI). spec.type could also be set to 1, which tells the AAD Pod Identity infrastructure that you want to use a Service Principal instead of an Azure Identity.

Once the identity is in place, it’s time to create a binding for it. The binding will be used to assign the identity to a range of Pods.

cat <<EOF | kubectl apply -f -
apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
  name: ${IDENTITY_NAME}-binding
spec:
  azureIdentity: ${IDENTITY_NAME}
  selector: ${IDENTITY_NAME}
EOF

In this yaml file, the Selector property is the most important one. It is used to identify which Pods should run in the context of the linked Azure Identity.

Repeat the above steps for the second identity for Vault by setting IDENTITY_NAME variable to vault.

Deploying Vault on AKS

Now let’s deploy Vault on AKS using the official Hashicorp Helm chart for Vault. We will use the following custom values for the Helm chart. In the values, we are setting the label aadpodidbinding to vault, which we have set as selector in the manifest when we defined AzureIdentityBinding. This will enable the Vault pods to use Vault Azure Identity.

$ cat <<EOF >>vault-values.yaml
server:
  standalone:
    enabled: true
    config: |
      ui = true
      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      storage "file" {
        path = "/vault/data"
      }
  service:
    enabled: true

  extraLabels:
    aadpodidbinding: "vault"

  dataStorage:
    enabled: true
    size: 10Gi
    storageClass: null
    accessMode: ReadWriteOnce

ui:
  enabled: true
  serviceType: LoadBalancer
  EOF

$ helm repo add hashicorp https://helm.releases.hashicorp.com
$ helm install vault hashicorp/vault --values vault-values.yaml

Now initialize and unseal Vault.

$ export LB_PUBLIC_IP=$(kubectl get svc vault-ui -o json | jq .status.loadBalancer.ingress[0].ip | sed 's/"//g')
$ export VAULT_SKIP_VERIFY=true
$ export VAULT_ADDR=http://LB_PUBLIC_IP:8200
$ vault operator init -key-shares 1 -key-threshold 1
$ vault operator unseal 

After Vault is initialized and unsealed, enable Azure auth method in Vault and create a role for AKS with a policy attached to it that will allow access to the KV store.

$ vault login
$ vault auth enable azure
$ vault write auth/azure/config tenant_id=$(az account show --query tenantId -o tsv) resource=https://management.azure.com/
$ vault write auth/azure/role/aks-role policies="aks" bound_subscription_ids=${SUBSCRIPTION_ID} bound_resource_groups=${IDENTITY_RESOURCE_GROUP}

Next enable a KV store and add a secret.

$ vault secrets enable -path=aks kv
$ vault kv put aks/mysecret foo=bar

Also, create the policy for this KV store.

$ vault policy write aks << EOF
path "aks/*" {
  capabilities = ["read", "list"]
}
EOF

Verify Azure Identity binding

Everything is in place now, so it’s time to deploy a demo pod to verify we can access secrets in Vault from this pod using AAD Pod Identity. To allow this pod to use our custom Azure Identity, we will add a label aaadpodidbinding with value demo in the pod specification. This value was set as selector property when we defined Azure Identity Binding.

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: demo
  labels:
    app: ubuntu
    aadpodidbinding: demo
spec:
  containers:
  - image: ubuntu
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: ubuntu
  restartPolicy: Always
EOF

From inside the pod session, let’s first grab the token from Azure Metadata service and then use the JWT token to authenticate with Vault.

$ kubectl get pods
$ kubectl exec -it demo -- bash

$ apt update && apt install jq curl -y
$ metadata=$(curl -H Metadata:true "http://169.254.169.254/metadata/instance?api-version=2019-08-15")

$ subscription_id=$(echo $metadata | jq -r .compute.subscriptionId)
$ vm_name=$(echo $metadata | jq -r .compute.name)
$ vmss_name=$(echo $metadata | jq -r .compute.vmScaleSetName)
$ resource_group_name=$(echo $metadata | jq -r .compute.resourceGroupName)

$ response=$(curl 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' -H Metadata:true -s)

$ jwt=$(echo $response | jq -r .access_token)

Now we will use the generated JWT token along with other parameters like subscription id, role name, resource group, vm name, etc. to authenticate with Vault.

$ cat <<EOF > auth_payload_complete.json
{
    "role": "aks-role",
    "jwt": "$jwt",
    "subscription_id": "$subscription_id",
    "resource_group_name": "$resource_group_name",
    "vm_name": "$vm_name",
    "vmss_name": "$vmss_name"
}
EOF

$ export VAULT_ADDR=http://LB_PUBLIC_IP:8200
$ VAULT_TOKEN=$(curl -s --request POST --data @auth_payload_complete.json $VAULT_ADDR/v1/auth/azure/login | jq .auth.client_token | sed 's/"//g')

At this stage, we have the Vault token and we can make request to Vault API to read the KV secret that we created.

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" $VAULT_ADDR/v1/aks/mysecret | jq .data.foo
"bar"

Conclusion

Azure AD Pod Identity allows you to bind Pods to an Azure Identity that is managed outside of your cluster. Vault can be configured to use Azure AD as identity provider, which will enable your containerized applications in AKS to consume secrets in Vault without any code modifications. I hope this workflow has provided a better understanding of the integration of Vault with Azure AKS using AAD Pod Identity. The AAD Pod Identity is an open source project and available here.