Rotating SSL Certificates With OpenShift and Spring Boot

1. Overview

OpenShift is a container application platform developed by the Red Hat Enterprise. It extends Kubernetes by adding custom components to enhance the developer experience. These components include a built-in CI/CD pipeline system, container image registry, service mesh, internal certificate authority (CA) service, and more.

In this tutorial, we’ll learn about the service CA operator in OpenShift and walk through a hands-on example to demonstrate the functionality of the service CA operator.

2. OpenShift’s Service CA Operator

To ensure data is encrypted during transmission between client and server, many services opt to serve their traffic over an SSL/TLS-secured channel. Essentially, both establish a secure communication channel before any data transmission occurs. For example, HTTPS is the HTTP protocol that runs over a secure channel. To enable an SSL/TLS secure channel, the server needs to have an SSL certificate along with the private and public key pairs.

2.1. SSL Certificate Management Requirements

SSL certificates usually don’t last long as a matter of best practice for security. Typically, the certificate lasts no longer than five years and must be rotated regularly. The lifecycle management for these certificates is often the source of problems in many organizations.

In a plain Kubernetes cluster, the management of the certificate is up to the administrator. Many opt for an external tool, such as cert-manager, to deal with the lifecycle of the certificates. In the OpenShift platform, there’s a built-in service CA operator that manages the lifecycle of the SSL certificates.

2.2. Service CA Operator

The service CA operator in OpenShift is a component that deals with digital certificates. Specifically, the operator creates a self-signed CA when an OpenShift cluster is deployed. Then, it issues and rotates the digital certificates for service resources that ask for them.

To support the functionalities, the service CA operator runs two controllers:

• serving cert signer
• configmap cabundle injector

Let’s briefly discuss both.

2.3. serving cert signer Controller

The serving cert signer is the controller for issuing a signed certificate to service resources annotated with service.beta.openshift.io/serving-cert-secret-name.

When we deploy a service resource with the annotation service.beta.openshift.io/serving-cert-secret-name=<secret-name>, the controller generates a certificate and respective key pairs signed by the service CA. Then, it stores the certificate, tls.crt, and private key, tls.key, into a Secret object with the same name as the annotation value.

The certificate generated lasts for two years before it expires. Additionally, the controller automatically rotates the certificate that it issues prior to its expiration.

2.4. configmap cabundle injector Controller

The configmap cabundle injector injects the CA bundle into the configmap resource with the annotation service.beta.openshift.io/inject-cabundle=true. If the configmap resource already contains a data key service-ca.crt, the controller updates the data with the service CA bundle. This makes the client code depend on the configmap resource to obtain the service CA bundle certificate.

3. OpenShift Service CA Operator in Action

In this section, we walk through a hands-on example that demonstrates how the service CA operator works.

3.1. Starting a CodeReady Container (CRC) Cluster

There are a few ways we can get access to an OpenShift cluster for development and testing. From version 4.0 onwards, we can create an OpenShift local cluster using the CodeReady Container (CRC) or MicroShift. For this demonstration, we use the CRC approach due to its simplicity and almost complete feature parity with the full-fledged OpenShift cluster.

To begin, we first follow the CRC official guide to install the CRC software suite on the system. Then, we start the cluster using the crc start command:

$ crc start
....
Started the OpenShift cluster
...

The console prints information about the OpenShift cluster we just created. We follow the instructions to set up the oc command to point to the new cluster:

$ eval $(crc oc-env)
$ oc login -u developer https://api.crc.testing:6443

Subsequently, let’s create a new project in the OpenShift cluster using the oc new-project command:

$ oc create-project ssl-rotate-demonstration
$ oc projects
You have one project on this server: "ssl-rotate-demonstration".

Using project "ssl-rotate-demonstration" on server "https://api.crc.testing:6443"

With the new project, the cluster is ready for application deployment.

3.2. Creating a Spring Boot Application With an HTTPS Endpoint

At this point, let’s create a Docker image of a Spring Boot application that exposes the /welcome endpoint through the HTTPS protocol. We can follow the guide for creating a Spring Boot HTTPS endpoint with a few exceptions.

First, we don’t need to generate the keystore bundle.

Additionally, we need to parameterize the path of the SSL keystore certificate and private key:

spring.ssl.bundle.pem:
  server:
    keystore:
      certificate: ${CERT_PATH}/tls.crt
      private-key: ${CERT_PATH}/tls.key

This configuration externalizes the path of the certificate and private key to a base path, CERT_PATH. Thus, we can specify this path later on the deployment resource manifest.

3.3. Deploying the Spring Boot Application

To deploy applications, we require the service and deployment resources in the OpenShift cluster.

Let’s create a manifest that defines the deployment resource:

$ cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ssl-rotate-demonstration-app
spec:
  selector:
    matchLabels:
      ssl-rotate-demonstration-app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ssl-rotate-demonstration-app
    spec:
      containers:
        - image: example/ssl-rotate-demonstration-app:latest
          name: ssl-rotate-demonstration-app
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: CERT_PATH
              value: /opt/ssl
          volumeMounts:
            - mountPath: /opt/ssl
              name: server-ssl-bundle
      volumes:
        - name: server-ssl-bundle
          secret:
            secretName: server-ssl-bundle

In the deployment manifest, we map the volume server-ssl-bundle to the secret object with the same name. This is the secret object that the service CA operator uses to store the generated SSL certificate and private key.

Then, we mount the volume on the path /opt/ssl. Finally, we complete the loop by setting the CERT_PATH to /opt/ssl. This way, the application should get the tls.crt and tls.key from the server-ssl-bundle secret object.

Then, we create the Service resource:

$ cat service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: ssl-rotate-demonstration-app
  annotations:
    service.beta.openshift.io/serving-cert-secret-name: server-ssl-bundle
  name: ssl-rotate-demonstration-service
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: ssl-rotate-demonstration-app
  type: ClusterIP

We annotate the service resource with the key service.beta.openshift.io/serving-cert-secret-name and the value server-ssl-bundle. With this annotation, the service CA operator creates a secret server-ssl-bundle that contains the generated SSL certificate and private key.

To deploy, we apply both manifests to the OpenShift clusters:

$ oc apply -f service.yaml
$ oc apply -f deployment.yaml

At this point, we should be ready. To confirm, let’s validate the setup.

3.4. Validating

To validate that the service CA operator actually creates the secret resource as expected, we can run the command oc describe secret server-ssl-bundle:

$ oc describe secret server-ssl-bundle
Name:         server-ssl-bundle
Namespace:    ssl-rotate-demonstration
Labels:       <none>
Annotations:  openshift.io/description:
                Secret contains a pair signed serving certificate/key that is generated by Service CA operator for service/ssl-rotate-demonstration-servic...
              openshift.io/owning-component: service-ca
              service.alpha.openshift.io/expiry: 2027-06-01T02:36:37Z
              service.beta.openshift.io/expiry: 2027-06-01T02:36:37Z
              service.beta.openshift.io/originating-service-name: ssl-rotate-demonstration-service
              service.beta.openshift.io/originating-service-uid: 96317ba8-a8af-40f3-9ef9-edb421aa0519

Type:  kubernetes.io/tls

Data
====
tls.crt:  2761 bytes
tls.key:  1679 bytes

We can observe a few things from the output:

1. The secret contains a description stating that it’s generated by the service CA operator as expected.
2. The annotation service.beta.openshift.io/expiry states the certificate’s expiration date, which is two years after the generation.
3. In the data section, we can see both the certificate file and private key under the tls.crt and tls.key, respectively.

This shows that the service CA operator generated the SSL certificate and stores it in a secret that we can subsequently reference in the application.

4. Manually Rotating SSL Certificate

The service CA operator should ensure the SSL certificates rotate regularly before expiration. However, there might be a time when we want to force-rotate the SSL certificate. For instance, when the certificate’s private key is compromised, we might want to refresh all SSL certificates.

To force rotate an SSL certificate, we can delete the secret object containing the SSL certificate:

$ oc delete secret server-ssl-bundle
secret "server-ssl-bundle" deleted

Immediately after we delete it, the service CA operator should recreate the secret with the new certificate and private key:

$ oc describe secret server-ssl-bundle
Name:         server-ssl-bundle
Namespace:    ssl-rotate-demonstration
Labels:       <none>
Annotations:  openshift.io/description:
                Secret contains a pair signed serving certificate/key that is generated by Service CA operator for service/ssl-rotate-demonstration-servic...
              openshift.io/owning-component: service-ca
              service.alpha.openshift.io/expiry: 2027-06-01T04:15:56Z
              service.beta.openshift.io/expiry: 2027-06-01T04:15:56Z
              service.beta.openshift.io/originating-service-name: ssl-rotate-demonstration-service
              service.beta.openshift.io/originating-service-uid: 96317ba8-a8af-40f3-9ef9-edb421aa0519

Type:  kubernetes.io/tls

Data
====
tls.crt:  2761 bytes
tls.key:  1679 bytes

Notably, the expiry date time is refreshed compared to when we first described the same secret object.

5. Conclusion

In this tutorial, we briefly learned that OpenShift builds on top of Kubernetes to offer more components out of the box that ease the development experience.

Then, we discussed the service CA operator and how its two controllers automate the management of SSL certificates. Finally, we walked through an example in which we deploy an HTTPS Spring Boot server application to an OpenShift cluster.