Secrets management is a critical aspect of modern infrastructure and application security. As applications grow more complex and distributed, effectively securing and distributing sensitive information like API keys, database credentials, and other secrets becomes increasingly challenging.

In this comprehensive guide, we'll explore how to leverage HashiCorp Vault and the External Secrets Operator (ESO) in Kubernetes to create a robust secrets management solution. This approach separates the concerns of secrets storage (Vault) from secrets consumption (Kubernetes), offering enhanced security and operational benefits.

What You'll Learn

• Core concepts of secrets management with Vault and Kubernetes

• How to set up a local Vault server using Docker

• How to configure the External Secrets Operator in Kubernetes

• How to create and manage secrets in Vault

• How to synchronize Vault secrets to Kubernetes using External Secrets

• How to verify and use the synchronized secrets

Prerequisites

• Docker and Docker Compose installed

• A Kubernetes cluster (we'll use Docker Desktop's built-in Kubernetes)

• kubectl command-line tool configured

• Helm package manager installed

The Architecture: How It All Works Together

Before diving into implementation, let's understand how these components work together:

1. HashiCorp Vault: A dedicated secrets management tool that securely stores sensitive data with encryption, access controls, and auditing capabilities.

2. External Secrets Operator (ESO): A Kubernetes operator that fetches secrets from external APIs and injects them as native Kubernetes Secret objects.

3. ClusterSecretStore: A Kubernetes custom resource that configures the connection between ESO and an external secrets provider (Vault in our case).

4. ExternalSecret: A Kubernetes custom resource that defines which secrets to fetch from the external provider and how to transform them into Kubernetes Secrets.

The workflow is:

• Store secrets in Vault using its rich management features

• Configure ESO to connect to Vault via a ClusterSecretStore

• Define ExternalSecret resources for specific secrets needed by applications

• ESO automatically creates and maintains corresponding Kubernetes Secrets

• Applications consume standard Kubernetes Secrets without any knowledge of Vault

This architecture provides several benefits:

• Separation of concerns: Security teams can manage the central secrets storage while application teams use familiar Kubernetes patterns

• Reduced risk surface: Access credentials to the central vault are only needed by ESO, not individual applications

• Automatic rotation: When secrets change in Vault, ESO can automatically update the Kubernetes Secrets

• Centralized audit trail: All secret access and modifications are logged in Vault

Implementation

Step 1: Setting Up Vault with Docker Compose

Let's start by creating a Docker Compose file to run Vault in development mode:

# docker-compose.yml
services:
  vault:
    image: hashicorp/vault:1.19.0
    container_name: vault
    ports:
      - "8200:8200"
    environment:
      - VAULT_DEV_ROOT_TOKEN_ID=root
      - VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200
    cap_add:
      - IPC_LOCK
    command

This configuration:

• Uses the official Vault Docker image

• Runs Vault in development mode (not for production use)

• Sets a predictable root token for ease of demonstration

• Exposes the Vault API on port 8200

• Adds the IPC_LOCK capability which Vault uses for memory locking

Start Vault:

docker compose up -d

Verify that Vault is running:

curl

Step 2: Setting Up Vault Secrets

Now let's set up a KV secrets engine in Vault and add some example secrets. We'll do this through the Vault UI for clarity, but you could also use the CLI or API.

1. Access the Vault UI at http://localhost:8200 and sign in with the token "root"

2. First, disable the default "secret" engine to start fresh:

   • Go to "secret/" in the left sidebar

   • Click the menu (three dots) and select "Disable"

   • Confirm by typing "secret" and clicking "Disable"

3. Create a new KV secrets engine:

   • Click "Enable new engine"

   • Select "KV" and click "Next"

   • Path: "org/kv"

   • Version: 2

   • Maximum number of versions: 3

   • Click "Enable Engine"

4. Create a development environment secret:

   • Navigate to your "org/kv" engine

   • Click "Create secret"

   • Path for this secret: "dev"

   • Secret data:

     • Key: "api-key"

     • Value: "123"

   • Click "Save"

5. Create a production environment secret:

   • Again, click "Create secret"

   • Path for this secret: "prod"

   • Secret data:

     • Key: "api-key"

     • Value: "456"

   • Click "Save"

You now have two secrets stored in Vault with the following paths:

• org/kv/dev

• org/kv/prod

Each contains an API key with different values for different environments.

Step 3: Installing External Secrets Operator in Kubernetes

Now let's set up the External Secrets Operator in our Kubernetes cluster:

# Add the External Secrets Helm repository
helm repo add external-secrets https://charts.external-secrets.io

# Update Helm repositories
helm repo update

# Install External Secrets Operator
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets \
  --create-namespace \
  --version 0

This installs version 0.15.0 of the operator with its custom resource definitions (CRDs) that we'll use to connect to Vault and define our external secrets. Using specific versions ensures consistent behavior and compatibility between components.

Wait for the pods to become ready:

kubectl get pods -n

You should see pods for the External Secrets Operator, the cert controller, and the webhook service.

Step 4: Creating Authentication for Vault

External Secrets Operator needs a way to authenticate with Vault. For our demo, we'll use a simple token-based authentication:

# Create a namespace for our demo
kubectl create namespace demo

# Create a Kubernetes secret with the Vault token
kubectl create secret generic vault-token \
  --namespace demo \
  --from-literal=token

This creates a Kubernetes secret that contains the Vault root token. In production, you would use a more limited token or a different authentication method like Kubernetes service account integration.

Step 5: Configuring the ClusterSecretStore

Next, we need to create a ClusterSecretStore resource that defines how ESO connects to Vault:

# secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://host.docker.internal:8200"
      path: "org/kv"
      version: "v2"
      auth:
        tokenSecretRef:
          name: "vault-token"
          namespace: "demo"
          key: "token"

This configuration:

• Uses the ClusterSecretStore CRD (which works across namespaces)

• Specifies the Vault server URL using host.docker.internal to access the host from within the Kubernetes cluster

• Sets the path to our secrets engine (org/kv)

• Specifies that we're using KV version 2

• Configures token-based authentication using the secret we created earlier

Apply this configuration:

kubectl apply -f

Step 6: Defining External Secrets

Now we can define ExternalSecret resources that specify which Vault secrets to sync to Kubernetes:

For the development API key:

# external-secret-dev.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: dev-api-key
  namespace: demo
spec:
  refreshInterval: "1m"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: dev-api-credentials
  data:
  - secretKey: api-key
    remoteRef:
      key: "dev"
      property: "api-key"

For the production API key:

# external-secret-prod.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: prod-api-key
  namespace: demo
spec:
  refreshInterval: "1m"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: prod-api-credentials
  data:
  - secretKey: api-key
    remoteRef:
      key: "prod"
      property: "api-key"

These configurations:

• Define two ExternalSecret resources, one for each environment

• Set a refresh interval so ESO checks for updates every minute

• Reference our ClusterSecretStore for connection details

• Specify the target Kubernetes Secret names that will be created

• Define the mapping between Vault secrets and Kubernetes Secret keys

Apply these configurations:

kubectl apply -f external-secret-dev.yaml
kubectl apply -f

Step 7: Verifying External Secret Synchronization

After applying the ExternalSecret resources, ESO will fetch the secrets from Vault and create corresponding Kubernetes Secrets. Let's verify this:

kubectl get secrets -n

You should see:

• vault-token (our authentication secret)

• dev-api-credentials (created by ESO)

• prod-api-credentials (created by ESO)

To check the content of the synchronized secrets:

kubectl get secret dev-api-credentials -n demo -o

You should see something like:

apiVersion: v1
kind: Secret
metadata:
  name: dev-api-credentials
  namespace: demo
  # Various metadata fields added by External Secrets Operator
type: Opaque
data:
  api-key: MTIz  # This is "123" base64 encoded

Similarly for the production secret:

kubectl get secret prod-api-credentials -n demo -o

To see the actual value:

kubectl get secret dev-api-credentials -n demo -o jsonpath='{.data.api-key}' | base64 --decode
echo

This should output 123, confirming that the secret has been correctly synchronized from Vault.

Step 8: Troubleshooting (If Needed)

When working with External Secrets Operator and Vault, you might encounter some common issues. Here are specific troubleshooting steps that have been tested with Vault v1.19.0 and ESO v0.15.0:

If you encounter issues with the external secrets not appearing, you can debug:

1. Check the status of the ExternalSecret:

   kubectl describe externalsecret dev-api-key -n

2. Look at the logs from the External Secrets Operator:

   kubectl logs -n external-secrets -l app.kubernetes.io/name

3. Verify your Vault paths using Vault itself:

   docker exec -it vault sh -c "VAULT_TOKEN=root VAULT_ADDR=http://127.0.0.1:8200 vault kv list org/kv"
   
   # To verify the actual secret content in Vault
   docker exec -it vault sh -c "VAULT_TOKEN=root VAULT_ADDR=http://127.0.0.1:8200 vault kv get org/kv/dev"
   
   # To check Vault's version
   
   

Step 9: Using Synchronized Secrets in Applications

Now that the secrets are synchronized to Kubernetes, pods can use them like any standard Kubernetes Secret. Here's an example:

# sample-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: api-consumer
  namespace: demo
spec:
  containers:
  - name: api-consumer
    image: busybox
    command: ['sh', '-c', 'echo "Using API Key: $API_KEY" && sleep 3600']
    env:
    - name: API_KEY
      valueFrom:
        secretKeyRef:
          name: dev-api-credentials
          key

Apply this Pod:

kubectl apply -f

Check that it's working:

kubectl logs api-consumer -n

You should see it output the API key from the synchronized secret.

Understanding the Value Proposition

This architecture provides several key benefits:

1. Centralized Management: All secrets are managed in Vault with its powerful security controls.

2. Decoupled Secrets Lifecycle: Applications don't need to be redeployed when secrets change.

3. Least Privilege: Applications only access the specific secrets they need.

4. Standardized Kubernetes Workflows: Applications use familiar Kubernetes Secret objects.

5. Automatic Updates: When secrets change in Vault, they're automatically updated in Kubernetes.

6. Audit Trail: All secret accesses are logged in a central location.

Conclusion

By combining HashiCorp Vault v1.19.0's powerful secrets management with the Kubernetes External Secrets Operator v0.15.0, we've built a system that provides the best of both worlds: centralized secrets management with distributed access using familiar Kubernetes patterns.

This approach enhances security by centralizing secrets storage and access control while simplifying application development by providing secrets through standard Kubernetes interfaces. The result is a more secure, maintainable, and operationally sound secrets management solution for your Kubernetes environments.

As you develop your own implementation, consider how this pattern can be extended to manage other types of secrets, integrate with CI/CD workflows, and adapt to your specific organizational requirements.

Kubernetes Training

If you find these guides helpful, check out our Kubernetes Training course