Sharing Secrets Across Namespaces in Kubernetes

Last updated: February 6, 2024

1. Overview

In Kubernetes, a Secret is an object that stores sensitive information such as API keys, credentials, and certificates. Passing sensitive information through the Secret object offers several benefits, such as limiting the number of actors who have access to the secret. However, the Secret object has a limitation whereby it is restricted to the namespace it is created in. This means we cannot easily refer to the same Secret object from another namespace.

In this tutorial, we’ll learn about creating Secret objects in Kubernetes and how to share the same Secret object across different namespaces.

2. Introduction to the Problem

To demonstrate the issue with passing Secret objects across the namespaces, we’ll be creating a Secret object and then attempting to refer to it from another namespace.

First, let’s look at a simple Secret definition:

$ cat app-secret.yaml 
apiVersion: v1 
kind: Secret 
metadata: 
  name: app-secret
type: Opaque
data: 
  secret_api_key: YWRtaW4=

The Secret definition above define the name of the Secret resource as app-secret, which is a unique identifier for this particular Secret resource. Then, it takes as input the secret key-value pairs through the data field. For our example, we’ve defined a secret key, secret_api_key, that stores the Base64-encoded value of admin. Note that Kubernetes is expecting the value of the secret to be Base64-encoded.

Then, we create the app-secret in the production namespace:

$ kubectl create namespace production
namespace/production created

$ kubectl apply -n production -f app-secret.yaml
secret/app-secret created

Next, let’s create a pod in the staging namespace that refers to the same app-secret:

$ cat deployment.yaml
apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: myapp
    image: alpine:latest
    command: ['env | grep SECRET_API_KEY']
    env:
    - name: SECRET_API_KEY
      valueFrom:
        secretKeyRef:
          name: app-secret
          key: secret_api_key

$ kubectl apply -n staging -f deployment.yaml
pod/myapp created

After running the command above, we’ll see that our pod fails to start up. Specifically, it’ll fail due to CreateContainerConfigError:

$ kubectl get pods -n staging
NAME    READY   STATUS                       RESTARTS   AGE
myapp   0/1     CreateContainerConfigError   0          68s

We can inspect the details using the kubectl describe command. The output of the command shows a lot of the details for the pod. For our demonstration, we’ll focus on the Event section:

$ kubectl describe -n staging pods myapp
...
 Warning  Failed     106s (x8 over 3m27s)  kubelet            Error: secret "app-secret" not found
...

From the event list, we can see that our app failed to start up because it couldn’t find the app-secret resource. This is because the app-secret is a resource in the production namespace and not the staging namespace.

Despite this limitation, there are several methods we can use to share the same Secret object across different namespaces. Let’s look at each method in detail.

3. Duplicating the Secret to Different Namespaces

The first method for sharing the secret among different namespaces is by simply creating the same secret on different namespaces. In other words, we run the kubectl apply command on our Secret manifest file for all the namespaces where we want the Secret to be available.

For example, let’s create the app-secret Secret object on our cluster:

$ kubectl apply -f app-secret.yaml
secret/app-secret created

By default, the Secret will be created at the default namespace. Then, we can recreate the same Secret on another namespace using the -n option on the kubectl command:

$ kubectl apply -f app-secret.yaml -n infrastructure
secret/app-secret created

$ kubectl get secrets -n infrastructure
NAME         TYPE     DATA   AGE
app-secret   Opaque   2      15s

To duplicate an existing Secret in the cluster, we can use a series of commands to apply it to a target namespace:

$ kubectl get secrets app-secret -n default -o json \
 | jq 'del(.metadata["namespace","creationTimestamp","resourceVersion","selfLink","uid","annotations"])' \
 | kubectl apply -n gateway -f -
secret/app-secret created

Let’s break this down. The one-liner above uses the kubectl get command to get the details of app-secret as a JSON string. Then, we remove all the unnecessary metadata from the output using the jq del command. Finally, we apply the resource using kubectl apply to create the Secret at the gateway namespace.

The advantage of this method is that it’s the easiest method with the least overhead. That’s because we don’t have to install any extension and the method is easily achievable using the kubectl command-line tool.

However, the downside is that there’s no synchronization mechanism between these different Secrets in different namespaces. In other words, whenever the Secret is changed, we’ll need to apply the same steps for all the namespaces. Therefore, if the Secret changes frequently, this method is not ideal.

Let’s look at the kubernetes-reflector extension that automatically synchronizes the different applications of the Secret for us.

4. Using kubernetes-reflector Extension

The kubernetes-reflector is a custom Kubernetes controller that replicates namespace resources across different namespaces in the cluster. Using the kubernetes-reflector, we can create a single Secret object and have the controller replicate it automatically. It also supports automatic synchronization of the Secret across different namespaces.

This extension is ideal for replicating the Secret objects that change frequently, such as rotating a short-term certificate.

4.1. Installation

To install the kubernetes-reflector extension in our cluster, we can apply the deployment file to our cluster:

$ kubectl apply -f https://github.com/emberstack/kubernetes-reflector/releases/latest/download/reflector.yaml

To ensure it’s up and running, we can check the status of the reflector Deployment object:

$ kubectl get deployments
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
reflector   1/1     1            1           3m6s

Once it’s ready, we can now start to replicate our Secret by adding some custom annotations.

4.2. Enabling Replication on Secrets

By default, the reflector will not replicate all the Secret objects in our cluster. To enable replication on our Secret objects, we’ll need to add at least two custom annotations to the Secret definition:

$ cat app-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: app-secret
  annotations:
    reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
    reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
type: Opaque
stringData:
  db_host: vendor-database.org
data:
  secret_api_key: YWRtaW4=

First, the reflector.v1.k8s.emberstack.com/reflection-allowed annotation enables the Secret object to be replicated by kubernetes-reflector. This is the bare minimum annotation we’ll have to set. Without this annotation, the reflector will not consider the Secret object as a resource to replicate.

Then, we also set the reflector.v1.k8s.emberstack.com/reflection-auto-enabled annotation to true. With this annotation, the Secret object will be automatically created on all the namespaces.

At the time of creation, all the existing namespaces will obtain a copy of the Secret. From there on, the kubernetes-reflector will react to any namespace creation event by replicating the Secret object to the new namespace.

After we’ve applied the definition, we can try it out by creating a new namespace:

$ kubectl create namespace test
namespace/test created

Then, let’s get the list of Secret objects in the test namespace:

$ kubectl get secrets -n test
NAME         TYPE     DATA   AGE
app-secret   Opaque   2      4s

Without our having to manually create the Secret on the new namespace, the controller automatically applies this new Secret to the new namespace. This is because we’ve set reflector.v1.k8s.emberstack.com/reflection-auto-enabled to true.

Besides replicating the Secret, the kubernetes-reflector also constantly monitors the source object to make sure the mirror objects are always in sync with the source.

Let’s change the secret_api_key secret value to a different value and apply it to the cluster. To do that, we’ll first read the content of the app-secret.yaml using the cat command-line tool. Then, we pipe the content of the file to the yq yaml editor and change the value of the secret_api_key. Finally, we apply the updated manifest to the cluster using kubectl apply:

$ cat app-secret.yaml \
 | yq e '.data.secret_api_key = "bmV3LXNlY3JldAo="' \
 | kubectl apply -f -
secret/app-secret configured

Now, let’s check the value of the replicated Secret in the test namespace:

$ kubectl get secrets app-secret -o jsonpath='{.data.secret_api_key}' -n test
bmV3LXNlY3JldAo=

As we can see from the output, the controller has also updated the replicated Secret when the source changed.

4.3. Restricting Replication to Certain Namespaces

We can further control the namespaces we want to replicate a Secret to using the reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces annotation. The annotation takes as input a list of comma-delimited strings of the target namespace names. For example, we can limit the replication of our secret to the test and staging namespaces only:

$ cat app-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: app-secret
  annotations:
    reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
    reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
    reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "test,staging"
type: Opaque
stringData:
  db_host: vendor-database.org
data:
  secret_api_key: YWRtaW4=

To test it out, let’s first create a different namespace, production, and then check its secrets:

$ kubectl create namespace production
namespace/production created

$ kubectl get secrets -n production
No resources found in production namespace.

As we can see from the output of the kubectl get secrets command, the kubernetes-reflector does not replicate the app-secret to the production namespace due to the namespace restriction annotation.

5. Conclusion

In this tutorial, we’ve learned that the Secret object in Kubernetes is the preferred way to distribute sensitive information, such as credentials, to applications. Then, we’ve seen that the Secret object resides in a namespace and is bound to a single one. Despite the limitation, we’ve looked at several methods to distribute the same Secret across different namespaces.

The first method simply creates the same Secret on all the namespaces, but the downside is that there’s no way to keep them in sync. Then, we learned about the kubernetes-reflector extension that solves the synchronization problem at the cost of managing an additional extension.

Comments are open for 30 days after publishing a post. For any issues past this date, use the Contact form on the site.