Skip to main content

Securing Hybrid Cloud Workloads with Google Anthos and Aqua Security

A year after writing the first blog on this topic here on using Aqua to provide comprehensive container security, a few things have started to change in the landscape of container development. Velocity has increased, automation tooling has gotten better, and serious [fully containerized] workloads are being put into production. With so much development taking place, companies are starting to build ephemeral/temporary kubernetes clusters for various reasons, while also running more than one production kubernetes cluster. This new form of infrastructure places a burden upon the operations teams in order to control and manage the many clusters that may be deployed with a desirable workload, and kubernetes configuration isn’t easy. Tools that assist in this world must have a centralized approach towards managing all of this ever-changing infrastructure.

One thing that’s clear though, is that as we layer multiple sets of tools and infrastructure to support these kubernetes clusters, our automation can become bloated, incredibly complex, and prone to error. There has to be a simpler way to manage these resources and still keep them secure.

With solutions like Google Anthos enabling a true hybrid cloud experience for kubernetes clusters, the burden of managing the configuration state of multiple kubernetes clusters can be drastically reduced with Config Management. Config Management is a component of the Google Anthos solution that enables centralized declarative configuration control with a GitOps approach. This means that a centralized git repository is responsible for managing the state of all “registered” clusters. It’s a one-stop-stop for config changes!

With a centralized configuration tool like Config Management, enterprise tools such as
Aqua Security can be deployed across all GKE or GKE On-Prem, and still have the desired granular control required depending on the type of resource that is needed to be secured. By using Config Management to control the deployment and configuration of Aqua Security, we can easily achieve:

  • CI/CD Scanning Integration for any cluster (permanent or ephemeral)
  • Automatic distribution of Enforcers across all GKE and GKE On-Prem nodes, providing:
    • Host vulnerability management
    • Image vulnerability management
    • Runtime enforcement
  • Centralized secrets management with distributed access, providing:
    • Real-time secret updates without pod restarts, and
    • Integration with secrets providers such as HashiCorp Vault
  • Centralized dashboards, providing:
    • Policy configuration
    • Audit logging
    • Cross-cluster visibility
  • Google Cloud Platform integration, providing;
    • Google Container Registry inventory and scanning
    • Google Stackdriver logging integration
    • Google SAML integration for user account management

High Level Overview of the Environment

{% include image name=”arch.png” position=”center” alt=”Architecture” %}

Deploying the Aqua security components with Anthos Config Management

Note: While the below content illustrates all Aqua Security configuration via the web console, the aquactl cli tool is a new addition to v4 and can be used for automating the configuration of your Aqua environment.

Ultimately, the daemonset is the most important component to place into Anthos Config Management, since it will ensure that all managed clusters receive Enforcers. However, for this lab, all components are deployed via Anthos Config Management.

In order to leverage Anthos Config Management to deploy the Aqua Security components:

  • A single aqua namespace was utilized to hold any applicable aqua components
  • All server/database/gateway components were deployed to a single cluster using using a ClusterSelector
  • 2 separate daemonset configurations were created, using a ClusterSelector, to target different GKE clusters with different Aqua Enforcer Policies

Let’s get going…

  • The aqua Config Management directory looks like the following:

│ ├── namespaces │ │ ├── aqua │ │ │ ├── csp-registry-secret.yml │ │ │ ├── db-deployment.yaml │ │ │ ├── db-password-secret.yaml │ │ │ ├── db-pvc.yaml │ │ │ ├── db-service.yaml │ │ │ ├── enforcer-daemonset-ephemeral-cluster.yml │ │ │ ├── enforcer-daemonset-gke-op-cluster.yml │ │ │ ├── gate-deployment.yaml │ │ │ ├── gate-service.yaml │ │ │ ├── namespace.yml │ │ │ ├── serviceaccount.yaml │ │ │ ├── web-deployment.yaml │ │ │ └── web-service.yaml
  • The server/database/gateway components are annotated with the proper cluster selector for the gke cluster with the label env: core (the labelling is arbitrarily decided by the operations team)

    • The metadata annotation to select the correct cluster
  • Separate daemonsets to align with different Enforcer groups

    • Ephemeral clusters are selected as follows
  • Long running production clusters are selected as follows:


Once the base configuration of the Aqua Server has been completed, create two separate Enforcer groups (generating the appropriate daemonset manifest)

{% include image name=”aqua_001.png” position=”center”%}

{% include image name=”aqua_002.png” position=”center” %}

Our two separate Enforcer groups now look like the following:

{% include image name=”aqua_003.png” position=”center”%}

Integrating with Google Container registry

Multiple container registry types are supported by Aqua Security, and Google Container registry is one such registry. The advantage of this integration is that Aqua can visualize and scan all images within the repository, not just the images that are scanned via the CI pipeline or images that are pulled down to worker nodes. This simple integration requires the service account json key or an access token to access the appropriate repository.

Under System → Integrations → Image Registries

{% include image name=”aqua_004.png” position=”center” alt=”Architecture” %}

And if Aqua is instructed to scan all images, the following dashboard will surface the relevant details:

{% include image name=”aqua_005.png” position=”center” %}

Integrating with CI & CD Pipelines

The story here hasn’t changed much regarding the capabilities of Aqua. Including scanning into the CI & CD pipeline is still a powerful feature with many options for integrations. In this demo environment, leveraging the Aqua Security Scanner Plugin for Jenkins is the defacto strategy, enabling developers to receive vulnerability feedback from any stage in the CI pipeline as outlined in this previous blog. The Aqua Microscanner is also a powerful tool for scanning on local builds or scanning during build process as outlined in this previous blog.

In this demo environment, an ephemeral GKE cluster is created, our application built and deployed, and then scanned for relevant results. The Enforcers are acting in Audit mode only.

{% include image name=”aqua_006.png” position=”center” %}

Centralized Secret Management with Distributed Access

Another advantage with running Aqua Enforcers in every GKE and GKE On-Prem cluster is that the Enforcers have access to any secrets managed by Aqua Security (and of course, secret access can have fine grained access policy applied to it). Aqua can serve up any secret as an environment variable or as a tmpfs file within a desired pod, regardless of the cluster or location, as long as an Enforcer is present. This functionality means that developers don’t need to create or synchronize secrets into each new cluster that they deploy, and furthermore, is often a simpler method of proxying secret access from a solution like Vault instead of deploying vault-agent sidecars or writing in direct API calls into the application.

Stay tuned for future blogs where we dig into the details of Secret distribution as well as additional Logging and Google SCC integrations with Aqua.

Interested in learning more about Anthos, Aqua Security or any of the topics I discussed?

We would love to hear from you.

//take the first step


Share this story

Arctiq Team

We service innovation.