Kubernetes Pods Storage and Volume Sharing

1. Introduction

The Kubernetes container orchestration platform provides several ways to transcend the usual ephemeral nature of containers. Mostly, we can create storage volumes that persist across pod and service restarts, resulting in consistent behavior and architectural redundancy. However, having different microservices use the same central volume is an option that Kubernetes offers natively only in part.

In this tutorial, we explore ways to share storage between different pods in a Kubernetes cluster. First, we briefly refresh our knowledge about data sharing and service architectures. After that, we turn to Kubernetes volumes to gain some understanding of types and access modes. Next, we show a basic example of a simple volume that can’t be shared beyond containers. Then, we demonstrate a Kubernetes storage type that can be shared between pods. Finally, we talk about a volume based on a Linux standard that can transcend container, pod, and node boundaries.

We tested the code in this tutorial on Debian 12 (Bookworm) with GNU Bash 5.2.15. Unless otherwise specified, it should work in most POSIX-compliant environments.

Despite the ability to share data between different readers and writers, doing so might not always be the best approach.

2.1. Data Views

Similar to the object-oriented programming (OOP) principle of keeping variables private and exposing them to other objects via interfaces and limited functions, microservices should usually attempt to encapsulate their data and preserve its integrity via special readers and writers.

This isn’t only related to security, but also integrity. All readers and writers of the same raw data should have an identical view of that data. If that view changes, each microservice should be able to handle the new structure.

On the other hand, when the data is exposed through interfaces, they can augment the new underlying structure to present the same view to each consumer.

Conversely, there are instances when a shared raw filesystem or path is inevitable or even desirable.

In such cases, we can resort to the mechanics offered by the platform that hosts the data, as well as the readers and writers.

3. Kubernetes Volumes

The Kubernetes platform provides volumes that we can attach to pods as a way to meet specific storage requirements.

There are many different types of volumes but they fall into two major categories:

ephemeral
persistent

One good example of an ephemeral volume is the emptyDir type, representing an empty directory local to the pod and containers within.

Still, we only consider persistent volumes due to the sharing requirement.

In addition to their persistence mode, volumes have many access modes:

ReadWriteOnce (RWO): read-write mountable by a single node
ReadWriteOncePod (RWOP): read-write mountable by a single pod
ReadOnlyMany (ROX): read-only mountable by many nodes
ReadWriteMany (RWX): read-write mountable by many nodes

Some volume types support multiple access modes. Still, they can only use one per mount.

Let’s see some of the main persistent volume types:

+----------------+---------------+--------------+---------------+------------------+
| Volume Type    | ReadWriteOnce | ReadOnlyMany | ReadWriteMany | ReadWriteOncePod |
+----------------+---------------+--------------+---------------+------------------+
| AzureFile      | v             | v            | v             | -                |
| CephFS         | v             | v            | v             | -                |
| FC             | v             | v            | -             | -                |
| FlexVolume     | v             | v            | -|v (driver)  | -                |
| HostPath       | v             | -            | -             | -                |
| iSCSI          | v             | v            | -             | -                |
| NFS            | v             | v            | v             | -                |
| RBD            | v             | v            | -             | -                |
| VsphereVolume  | v             | -            | -             | -                |
| PortworxVolume | v             | -            | v             | -                |
+----------------+---------------+--------------+---------------+------------------+

While HostPath is a convenient volume type that shares a path on the host with all local pods, it doesn’t work between nodes.

Of course, we don’t expect the ReadWriteOncePod access mode for persistent volumes. Then again, only four or five of the other types support ReadWriteMany.

Of these, NFS is usually the most well-known and convenient.

4. Kubernetes emptyDir Volume

To begin with, let’s explore a simple case of volume usage via the run subcommand and some –overrides:

$ kubectl run -i pod0 --image=debian:latest --overrides='{
  "apiVersion": "v1",
  "spec": {
    "containers": [
      {
        "name": "pod0",
        "image": "debian:latest",
        "command": ["bash"],
        "stdin": true,
        "stdinOnce": true,
        "tty": true,
        "volumeMounts": [
          {
            "mountPath": "/home/emptydir-vol0-attach-path",
            "name": "emptydir-vol0"
          }
        ]
      }
    ],
    "volumes": [
      {
        "name": "emptydir-vol0",
        "emptyDir": {}
      }
    ]
  }
}'

Here, we create the pod0 pod based on the latest Debian. It only runs Bash, enabling direct std[-i]n access and keeping the container running.

Notably, we also set up emptydir-vol0 as a simple emptyDir volume with its basic characteristics:

created after node assignment
shared between containers within the pod
can be under different paths for each container
persists between container restarts
destroyed upon node unassignment, i.e., pod destruction

Naturally, emptydir-vol0 files can be read and written by each container of the same pod. Still, an emptyDir volume won’t be available to other pods, even if we create a pod with the same settings.

5. Kubernetes hostPath Volume

One of the most natural persistent volume types that Kubernetes offers is hostPath:

$ kubectl run -i --rm pod0 --image=debian:latest --overrides='{
  "apiVersion": "v1",
  "spec": {
    "containers": [
      {
        "name": "pod0",
        "image": "debian:latest",
        "command": ["bash"],
        "stdin": true,
        "stdinOnce": true,
        "tty": true,
        "volumeMounts": [
          {
            "mountPath": "/home/hostpath-vol0-attach-path",
            "name": "hostpath-vol0"
          }
        ]
      }
    ],
    "volumes": [
      {
        "name": "hostpath-vol0",
        "hostPath": {
          "path": "/mnt/hostpath-vol0-source-path",
          "type": "DirectoryOrCreate"
        }
      }
    ]
  }
}'

In this case, hostpath-vol0 is a hostPath volume of the DirectoryOrCreate type. This means that the /mnt/hostpath-vol0-source-path/ path on the pod host node is directly used as a directory with Kubernetes creating it if it doesn’t already exist. Thus, /home/hostpath-vol0-attach-path/ on the pod maps directly to /mnt/hostpath-vol0-source-path/ on the host.

This enables direct data sharing between the hosting node and the pod. Further, we can use the same path with hostPath volumes on other pods, thereby attaching other readers and writers to the same data.

For instance, let’s create pod0 and pod1 with identical settings:

$ for podname in pod0 pod1; do
  kubectl run $podname --image=debian:latest --overrides='{
    "apiVersion": "v1",
    "spec": {
      "containers": [
        {
          "name": "'$podname'",
          "image": "debian:latest",
          "command": ["bash"],
          "stdin": true,
          "stdinOnce": true,
          "tty": true,
          "volumeMounts": [
            {
              "mountPath": "/home/hostpath-vol0-attach-path",
              "name": "hostpath-vol0"
            },
            {
              "mountPath": "/home/emptydir-vol0-attach-path",
              "name": "emptydir-vol0"
            }
          ]
        }
      ],
      "volumes": [
        {
          "name": "hostpath-vol0",
          "hostPath": {
            "path": "/mnt/hostpath-vol0-source-path",
            "type": "DirectoryOrCreate"
          }
        },
        {
          "name": "emptydir-vol0",
          "emptyDir": {}
        }
      ]
    }
  }'
done

Now, we can write a new hostfile file to the /mnt/hostpath-vol0-source-path/ host path on the node that runs the pods:

$ echo 666 > /mnt/hostpath-vol0-source-path/hostfile

In fact, we can also do the same with any file from either of the pods via exec:

$ kubectl exec -it pod0 -- bash -c 'echo 667 > /home/hostpath-vol0-attach-path/podfile'

Next, we check the contents of /home/hostpath-vol0-attach-path/hostfile for each pod:

$ for podname in pod0 pod1; do
  kubectl exec -it $podname -- bash -c 'cat /home/hostpath-vol0-attach-path/hostfile; cat /home/hostpath-vol0-attach-path/podfile'
done
666
667
666
667

As expected, the same data appears for both.

However, if we write data under the /home/emptydir-vol0-attach-path/ emptyDir path of pod0, we won’t see it in pod1 and vice-versa:

$ kubectl exec -it pod0 -- bash -c 'echo 668 &gt; /home/emptydir-vol0-attach-path/pod0file'
$ kubectl exec -it pod1 -- bash -c 'cat /home/emptydir-vol0-attach-path/pod0file'
cat: /home/emptydir-vol0-attach-path/pod0file: No such file or directory
command terminated with exit code 1

As expected, the emptydir-vol0 attachment paths aren’t shared between pods. Despite the identical settings for the emptyDir volumes, they are entirely different entities.

Still, even using hostPath volumes is strongly discouraged as it may present a security risk. Further, it doesn’t support the ReadWriteMany access mode, i.e., a hostPath volume can’t be used outside the local node.

6. Kubernetes and the Network File System (NFS)

As usual, when it comes to data sharing in a Linux environment, we can fall back to the standard Network FileSystem (NFS). Specifically, we can set up an NFS server and use it as a volume from any pod.

There are several advantages to this approach:

flexible method
standard protocol
well-tested mechanism
good performance

Of course, there are downsides as well:

protocol limitations
using an external toolset

To employ NFS to share data between pods, we can use a node or pod as the server, or even point clients to an external NFS server.

So, let’s create a pod that creates and uses a persistent NFS volume:

$ kubectl run pod0 --image=debian:latest --overrides='{
  "apiVersion": "v1",
  "spec": {
    "containers": [
      {
        "name": "pod0",
        "image": "debian:latest",
        "command": ["bash"],
        "stdin": true,
        "stdinOnce": true,
        "tty": true,
        "volumeMounts": [
          {
            "mountPath": "/home/nfs-vol0-attach-path",
            "name": "nfs-vol0"
          }
        ]
      }
    ],
    "volumes": [
      {
        "name": "nfs-vol0",
        "nfs": {
          "server": "192.168.6.60",
          "path": "/mnt/nfs/"
        }
      }
    ]
  }
}'
pod/pod0 created

In this case, the nfs-vol0 volume connects to an NFS server at 192.168.6.60.

Now, we can perform a brief test.

First, we write some data to a file locally on the NFS server:

$ echo 666 > /mnt/nfs/file

After that, we check what the pod sees under /home/nfs-vol0-attach-path/:

$ kubectl exec -it pod0 -- cat /home/nfs-vol0-attach-path/file
666

As expected, we see the same data.

7. Summary

In this article, we explored ways to get storage shared between Kubernetes pods.

In conclusion, although there are many types of volumes, only some can be used by more than one pod simultaneously, and even fewer – by pods on different nodes.

Full Archive

About Baeldung