Secure Kubernetes Secrets with Vault

1. Introduction

In this tutorial, we’ll explore different ways to access secrets stored in Hashicorp’s Vault from an application running on Kubernetes.

2. Quick Recap

We’ve already covered Hashicorp’s Vault in earlier tutorials, where we’ve shown how to install and populate it with secrets. In a nutshell, Vault provides a secure storage service for application secrets, which can be either static or dynamically generated.

To access Vault services, an application must authenticate itself using one of the available mechanisms. When the application runs in a Kubernetes environment, Vault can authenticate it based on its associated service account, thus eliminating the need for separate credentials.

In this scenario, the Kubernetes service account is bound to a Vault role, which defines the associated access policy. This policy defines which secrets an application can have access to.

3. Providing Secrets to Applications

In Kubernetes environments, a developer has multiple options to get a Vault-managed secret, which can be classified as more or less intrusive. “Intrusive”, in this context, relates to the level of awareness the application has about the origin of the secrets.

Here’s a summary of the methods we’ll cover:

• Explicit retrieval using Vault’s API
• Semi-explicit retrieval using Spring Boot’s Vault support
• Transparent support using Vault Sidecar
• Transparent support using Vault Secret CSI Provider
• Transparent support using Vault Secret Operator

4. Authentication Setup

In all those methods, the test application will use Kubernetes authentication to access Vault’s API. When running inside Kubernetes, this would be automatically provided. However, to use this kind of authentication from outside the cluster, we need a valid token associated with a service account.

One way to achieve this is to create a service account token secret. Secrets and service accounts are namespace-scoped resources, so let’s start by creating a namespace to hold them:

$ kubectl create namespace baeldung

Next, we create the service account:

$ kubectl create serviceaccount --namespace baeldung vault-test-sa

Finally, let’s generate a token valid for 24 hours and save it to a file:

$ kubectl create token --namespace baeldung vault-test-sa --duration 24h > sa-token.txt

Now, we need to bind the Kubernetes service account with a Vault role:

$ vault write auth/kubernetes/role/baeldung-test-role \
  bound_service_account_names=vault-test-sa \
  bound_service_account_namespaces=baeldung \
  policies=default,baeldung-test-policy \
  ttl=1h

5. Explicit Retrieval

In this scenario, the application gets the required secrets directly using Vault’s REST API or, more likely, using one of the available libraries. For Java, we’ll use the  library from the spring-vault project, which leverages the Spring Framework for low-level REST operations:

<dependency>
    <groupId>org.springframework.vault</groupId>
    <artifactId>spring-vault-core</artifactId>
    <version>3.0.4</version>
</dependency>

The latest version of this dependency is available on Maven Central.

Please make sure to pick a version that is compatible with Spring Framework’s main version: spring-vault-core 3.x requires Spring 6.x, while spring-vault-core 2.x requires Spring 5.3.x.

The main entry point to access Vault’a API is the VaultTemplate class. The library provides the EnvironmentVaultConfiguration helper class that simplifies the process of configuring a VaultTemplate instance with the required access and authentication details. To use it, the recommended way is to import it from one of the @Configuration classes of our application:

@Configuration
@PropertySource("vault-config-k8s.properties")
@Import(EnvironmentVaultConfiguration.class)
public class VaultConfig {
    // No code!
}

In this case, we’re also adding the vault-config-k8s property source, where we’ll add the required connection details. At a minimum, we need to inform Vault’s endpoint URI and authentication mechanism to use. Since we’ll be running our application outside of a cluster during development, we also need to supply the location of the file holding the service account token:

vault.uri=http://localhost:8200
vault.authentication=KUBERNETES
vault.kubernetes.role=baeldung-test-role
vault.kubernetes.service-account-token-file=sa-token.txt

We can now inject a VaultTemplate wherever we need to access Vault’s API. As a quick example, let’s create a CommandLineRunner @Bean that lists the contents of all secrets:

@Bean
CommandLineRunner listSecrets(VaultTemplate vault) {

    return args -> {
        VaultKeyValueOperations ops = vault.opsForKeyValue("secrets", VaultKeyValueOperationsSupport.KeyValueBackend.KV_2);
        List<String> secrets = ops.list("");
        if (secrets == null) {
            System.out.println("No secrets found");
            return;
        }

        secrets.forEach(s -> {
            System.out.println("secret=" + s);
            var response = ops.get(s);
            var data = response.getRequiredData();

            data.entrySet()
              .forEach(e -> {
                  System.out.println("- key=" + e.getKey() + " => " + e.getValue());
              });
        });
    };
}

In our case, Vault has a KV Version 2 secrets engine mounted on /secrets path, so we’ve used the opsForKeyValue method to get a VaultKeyValueOperations object that we’ll use to list all secrets. Other secret engines also have dedicated operations objects that offer tailor-made methods to access them.

For secret engines that do not have a dedicated VaultXYZOperations façade, we can use generic methods to access any path:

• read(path): Reads data from the specified path
• write(path, data): Writes data at the specified path
• list(path): Returns a list of entries under the specified path
• delete(path): Remove the secret at the specified path

6. Semi-Explicit Retrieval

In the previous method, the fact that we’re directly accessing Vault’a API introduces a strong coupling that may pose a few hurdles. For instance, this means developers will need a Vault instance or create mocks during development time and when running CI pipelines.

Alternatively, we can use the Spring Cloud Vault library in our projects to make Vault’s secret lookup transparent to the application’s code. This library makes this possible by exposing a custom PropertySource to Spring, which will be picked up and configured during the application bootstrap.

We call this method “semi-explicit” because, while it is true that the application’s code is unaware of Vault’s usage, we still must add the required dependencies to the project. The easiest way to achieve this is by using the available starter library:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-vault-config</artifactId>
    <version>3.1.3</version>
</dependency>

The latest version of this dependency is available on Maven Central.

As before, we must pick a version compatible with Spring Boot’s main version used by our project. Spring Boot 2.7.x requires version 3.1.x, while Spring Boot 3.x requires version 4.x.

To enable Vault as a property source, we must add a few configuration properties. A common practice is to use a dedicated Spring profile for this, which allows us to switch from Vault-based secrets to any other source quickly.

For Kubernetes, this is what a typical configuration properties file looks like:

spring.config.import=vault://
spring.cloud.vault.uri=http://vault-internal.vault.svc.cluster.local:8200
spring.cloud.vault.authentication=KUBERNETES
spring.cloud.vault.kv.backend=secrets
spring.cloud.vault.kv.application-name=baeldung-test

This configuration enables Vault’s KV backend to be mounted at the secrets path on the server. The library will use the configured application name as the path under this backend from where to pick secrets.

The spring.config.import property is also needed to enable Vault as a property source. Please notice this property was introduced in Spring Boot 2.4, along with the deprecation of the bootstrap context initialization. When migrating applications based on older versions of Spring Boot, this is something to pay special attention to.

The complete list of available configuration properties is available on Spring Cloud Vault’s documentation.

Now, let’s show how to use this with a simple example that gets a configuration value from Spring’s Environment:

@Bean
CommandLineRunner listSecrets(Environment env) {
    return args -> {
        var foo = env.getProperty("foo");
        Assert.notNull(foo, "foo must have a value");
        System.out.println("foo=" + foo);
    };
}

When we run this application, we can see the secret’s value in the output, confirming that the integration is working.

7. Transparent Support Using Vault Sidecar

If we don’t want to or can’t change the code of an existing application to get its secrets from Vault, using Vault’s sidecar method is a suitable alternative. The only requirement is that the application is already capable of picking values from environment variables and configuration files.

The sidecar pattern is a common practice in the Kubernetes landscape, where an application delegates some specific functionality to another container running on the same pod. A popular application of this pattern is the Istio service mesh, used to add traffic control policies and service discovery, among other functionalities, to existing applications.

We can use this approach with any Kubernetes workload type, such as a Deployment, Statefulset, or Job. Moreover, we can use a Mutating Webhook to automatically inject a sidecar when a pod is created, thus relieving users from manually adding it to the workload’s specification.

The Vault sidecar uses annotations present in the metadata section of the workload’s pods template to instruct the sidecar which secrets to pull from Vault. Those secrets are then put in a file stored in a shared volume between the sidecar and any other container in the same pod. If any of those secrets is dynamic, the sidecar also takes care of keeping track of its renewal, re-rendering the file when needed.

7.1. Sidecar Injector Deployment

Before we use this method, firstly, we need to deploy Vault’s Sidecar Injector component. The easiest way to do it is to use Hashicorp’s provided helm chart, which, by default, already adds the injector as part of regular Vault deployment on Kubernetes.

If this is not the case, we must upgrade the existing helm release with a new value for the injector.enabled property:

$ helm upgrade vault hashicorp/vault -n vault --set injector.enabled=true

To verify that the injector was properly installed, let’s query the available WebHookConfiguration objects:

$ kubectl get mutatingwebhookconfiguration
NAME                       WEBHOOKS   AGE
vault-agent-injector-cfg   1          16d

7.2. Annotating Deployments

A secret injection is “opt-in”, meaning that no changes will occur unless the injector finds specific annotations as part of a workload’s metadata. This is an example of a deployment manifest using the minimal set of required annotations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: baeldung
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1 
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-baeldung.properties: "secrets/baeldung-test"
        vault.hashicorp.com/role: "baeldung-test-role"
        vault.hashicorp.com/agent-inject-template-baeldung.properties: |
          {{- with secret "secrets/baeldung-test" -}}
          {{- range $k, $v := .Data.data }}
          {{$k}}={{$v}}
          {{- end -}}
          {{ end }}
    spec:
      serviceAccountName: vault-test-sa
      automountServiceAccountToken: true
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

When we deploy this manifest to the cluster, the injector will patch it and inject the Vault agent sidecar container configured as follows:

• Auto-login using baeldung-test-role
• Secrets located under the secrets/baeldung-test path will be renderer to a file named baeldung.properties under the default secret directory (/vault/secrets)
• File content generated using the provided template

There are many more annotations available, which we can use to customize the location and template used to render the secrets. The full list of supported annotations is available on Vault’s documentation.

8. Transparent Support Using Vault Secret CSI Provider

A CSI (Container Storage Interface) provider allows a vendor to extend the types of volumes supported by a Kubernetes cluster. The Vault CSI Provider is an alternative to the use of a sidecar that allows Vault secrets to be exposed to a pod as a regular volume.

The main advantage here is that we don’t have a sidecar attached to each pod, so we need fewer resources (CPU/Memory) to run our workload. While not very resource-hungry, the sidecar cost scales with the number of active pods. In contrast, the CSI uses a DaemonSet, which means there’s one pod for each node in the cluster.

8.1. Enabling the Vault CSI Provider

Before we can install this provider, we must check whether the CSI Secret Store Driver is already present in the target cluster:

$ kubectl get csidrivers

The result should include the secrets-store.csi.k8s.io driver:

NAME                       ATTACHREQUIRED   PODINFOONMOUNT  ... 
secrets-store.csi.k8s.io   false            true             ...

If this is not the case, it’s just a matter of applying the appropriate helm chart:

$ helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
$ helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
     --namespace kube-system\
     --set syncSecret.enabled=true

The project’s documentation also describes other installation methods, but unless there are some specific requirements, the helm method is the preferred one.

Now, let’s move on to the Vault CSI Provider installation itself. Once again, we’ll use the official Vault helm chart. The CSI provider is not enabled by default, so we need to upgrade it using the csi.enabled property:

$ helm upgrade vault hashicorp/vault -n vault –-set csi.enabled=true

To verify that the driver was correctly installed, we’ll check that its DaemonSet is running fine:

$ kubectl get daemonsets –n vault
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
vault-csi-provider 1 1 1 1 1 <none> 15d 

8.2. Vault CSI Provider Usage

Configuring a workload with vault secrets using the Vault CSI Provider requires two steps. Firstly, we define a SecretProviderClass resource that specifies the secret and keys to retrieve:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: baeldung-csi-secrets
  namespace: baeldung
spec:
  provider: vault
  parameters:
    roleName: 'baeldung-test-role'
    objects: |
      - objectName: 'baeldung.properties'
        secretPath: "secrets/data/baeldung-test"

Notice the spec.provider property, which must be set to vault. This is required so the CSI Driver knows which of the available providers to use. The parameters section contains information used by the provider to locate the requested secrets:

• roleName: Vault role used during login, which defines the secrets the application will have access to
• objects: The value is a YAML-formatted string (hence the “|”) with an array of secrets to retrieve

Each entry in the objects array is an object with three properties:

• secretPath: Vault’s path for the secret
• objectName: Name of the file that will contain the secret
• objectKey: Key within Vault’s secret that provides the content to put into the file. If omitted, the file will contain a JSON object with all values

Now, let’s use this resource in a sample deployment workload:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-csi
  namespace: baeldung
spec:
  selector:
    matchLabels:
      app: nginx-csi
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx-csi
    spec:
      serviceAccountName: vault-test-sa
      automountServiceAccountToken: true
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        volumeMounts:
        - name: vault-secrets
          mountPath: /vault/secrets
          readOnly: true
      volumes:
      - name: vault-secrets
        csi:
          driver: 'secrets-store.csi.k8s.io'
          readOnly: true
          volumeAttributes:
            secretProviderClass: baeldung-csi-secrets

In the volumes section, notice how we use a CSI definition pointing to our previously defined SecretStorageClass.

To validate this deployment, we can open a shell into the main container and check the presence of the secret under the specified mount path:

$ kubectl get pods -n baeldung -l app=nginx-csi
NAME                        READY   STATUS    RESTARTS   AGE
nginx-csi-b7866bc69-njzff   1/1     Running   0          19m
$ kubectl exec -it -n baeldung nginx-csi-b7866bc69-njzff -- /bin/sh
# cat /vault/secrets/baeldung.properties
{"request_id":"eb417a64-b1c4-087d-a5f4-30229f27aba1","lease_id":"","lease_duration":0,
   "renewable":false,
   "data":{
      "data":{"foo":"bar"},
  ... more data omitted  

9. Transparent Support Using Vault Secrets Operator

The Vault Secrets Operator adds Custom Resource Definitions (CRDs) to a Kubernetes cluster, which we can use to populate regular secrets with values pulled from a Vault instance.

Compared to the CSI method, the operator’s main advantage is that we don’t need to change anything on existing workloads to move from standard secrets to Vault-backed ones.

9.1. Vault Secrets Operator Deployment

The operator has its own chart that deploys all required artifacts into the cluster:

$ helm install --create-namespace --namespace vault-secrets-operator \
    vault-secrets-operator hashicorp/vault-secrets-operator \
    --version 0.1.0

Now, let’s check the new CRDs:

$ kubectl get customresourcedefinitions | grep vault
vaultauths.secrets.hashicorp.com                            2023-09-13T01:08:11Z
vaultconnections.secrets.hashicorp.com                      2023-09-13T01:08:11Z
vaultdynamicsecrets.secrets.hashicorp.com                   2023-09-13T01:08:11Z
vaultpkisecrets.secrets.hashicorp.com                       2023-09-13T01:08:11Z
vaultstaticsecrets.secrets.hashicorp.com                    2023-09-13T01:08:11Z

As of this writing, the operator defines those CRDs:

• VaultConnection: Defines Vault connection details such as its address, TLS certificates, etc
• VaultAuth: Authentication details used by a specific VaultConnection
• Vault<type>Secret: Defines a mapping between Kubernetes and Vault secrets, where <type> can be Static, Dynamic, or PKI, and corresponds to the secret type.

9.2. Vault Secrets Operator Usage

Let’s walk through a simple example to show how to use this Operator. Firstly, we need to create a VaultConnection resource pointing to our Vault instance:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  namespace: baeldung
  name: vault-local
spec:
  address: http://vault.vault.svc.cluster.local:8200

Next, we need a VaultAuth resource with the authentication details we’ll use to access secrets:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  namespace: baeldung
  name: baeldung-test
spec:
  vaultConnectionRef: vault-local
  method: kubernetes
  mount: kubernetes
  Kubernetes:
    role: baeldung-test-role
    serviceAccount: vault-test-sa

Those are the key properties we must fill in:

• spec.vaultConnectionRef: name of the VaultConnection resource we’ve just created
• spec.method: set to kubernetes as we’ll use this authentication method
• spec.kubernetes.role: Vault role to use when authenticating
• spec.kubernetes.serviceAccount: Service account to use when authenticating

Now, let’s define a VaultStaticSecret to map secrets from secrets/baeldung-test on Vault to a secret named baeldung-test:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  namespace: baeldung
  name: baeldung-test
spec:
  vaultAuthRef: baeldung-test
  mount: secrets
  type: kv-v2
  path: baeldung-test
  refreshAfter: 60s
  hmacSecretData: true
  destination:
    create: true
    name: baeldung-test

Finally, we can use kubectl to confirm that the secret was correctly created:

$ kubectl get secret -n baeldung baeldung-test
NAME            TYPE     DATA   AGE
baeldung-test   Opaque   3      24h

10. Method Comparison

As we’ve seen, Kubernetes-based applications have no shortage of alternatives to access secrets from Vault. To help choose which one is best suited for a given use case, we’ve put together a short comparison of each method’s features/characteristics:

11. Conclusion

In this tutorial, we’ve explored different ways to access secrets stored in a Vault instance from Kubernetes-based applications.

As always, all code is available over on GitHub.