How to Find the Space Left in a Persistent Volume Claim in Kubernetes

1. Introduction

In Kubernetes, a Persistent Volume Claim (PVC) is used to request storage resources from the cluster. When a PVC is created, it’s bound to a Persistent Volume (PV) that provides the actual storage. As applications run and data is written to the PVC, the available space on the underlying PV decreases. It’s important to monitor the remaining space in a PVC to ensure that applications don’t run out of storage and cause disruptions.

In this tutorial, we’ll explore different methods to find the space left in a PVC in Kubernetes.

2. Using kubectl

kubectl is a command-line tool for interacting with Kubernetes clusters. It’s the primary way for managing Kubernetes clusters, including deploying applications, inspecting and managing cluster resources, debugging, and troubleshooting.

2.1. Getting a List of PVCs

First, we get a list of all the PVCs in the cluster or a specific namespace:

$ kubectl get pvc --all-namespaces
NAMESPACE      NAME        STATUS   VOLUME    CAPACITY   ACCESS MODES   AGE
default        data-pvc    Bound    data      5Gi        RWO            12d
kube-system    logs-pvc    Bound    logs      10Gi       RWO            8d

This command displays a list of all PVCs in the cluster, along with their names, status, volume, capacity, access modes, and age.

2.2. Describe a Specific PVC

Further, we use the describe option to get detailed information about a specific PVC by name:

$ kubectl describe pvc data-pvc -n default
Name:          data-pvc
Namespace:     default
StorageClass:  standard
Status:        Bound
Volume:        data
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      5Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    <none>
Events:        <none>

In this case, we employ the kubectl describe pvc command, the Persistent Volume Claim data-pvc name, and the namespace default to retrieve detailed information about the PVC.

Now, let’s inspect the Pods using the PVC and check their filesystems.

Thus, we list the Pods that use the PVC:

$ kubectl get pods -n default --selector=pvc=data-pvc
NAME       READY   STATUS    RESTARTS   AGE
data-pod-1   1/1     Running   0          10d
data-pod-2   1/1     Running   0          5d

As we can see, the output lists two Pods, data-pod-1 and data-pod-2.

2.3. Check Utilized Space via an exec Shell

At this point, we can select one of these Pods and enter it via kubectl exec and the respective shell path:

$ kubectl exec -it data-pod-1 -n default -- /bin/bash

Once connected, we use the du command to check the utilization of the PVC’s filesystem, depending on the path name:

$ du -sh /mnt/data
3.0Gi /mnt/data

Then, we repeat the same steps for all other Pods to get their utilization, and finally, we subtract the total utilized space from the total capacity of the PVC to find the space left.

2.4. Check Utilized Space via exec and df

Alternatively, we can execute a command within the Pod’s container and check the available disk space.

Notably, this method doesn’t work if the container doesn’t have the df command or the Pod cannot start due to an error.

For instance, we can execute the tool for data-pod-1:

$ kubectl -n default exec data-pod-1 -- df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       5Gi   3Gi   2Gi  60% /mnt/data

Let’s break this command down:

• -n specifies the namespace where the Pod is located
• exec executes a command inside a running container
• df -h is the command that runs inside the Pod

In this instance, Kubernetes executes the df -h command inside the data-pod-1 Pod in the default namespace. Thus, we get information about the storage space usage in the file systems associated with the persistent mounted volume within that Pod.

3. Query Prometheus Metrics

Querying Prometheus metrics refers to accessing and retrieving metrics data from a Prometheus server or endpoint. After that, we can use these metrics to obtain real-time information about the PVC capacity and utilization.

The kubelet_volume_stats_available_bytes and kubelet_volume_stats_capacity_bytes are metrics provided by kubelet, the primary node agent running on each node in the Kubernetes cluster. Therefore, we can use them to measure the space left in a persistent volume claim in Kubernetes.

Let’s take a look at the script:

$ cat metrics.sh
#!/bin/bash

# Retrieve available space in bytes
available_bytes=$(curl -s http://kubelet-address:10255/metrics | grep kubelet_volume_stats_available_bytes | grep my-pvc | awk '{print $2}')

# Retrieve total capacity in bytes
capacity_bytes=$(curl -s http://kubelet-address:10255/metrics | grep kubelet_volume_stats_capacity_bytes | grep my-pvc | awk '{print $2}')

# Convert bytes to gigabytes (GB)
available_gb=$(echo "scale=2; $available_bytes / 1024^3" | bc)
capacity_gb=$(echo "scale=2; $capacity_bytes / 1024^3" | bc)

# Display the results
echo "Available space: $available_gb GB"
echo "Total capacity: $capacity_gb GB"

In detail, we use curl to get metrics data for the specified PVC from the kubelet Prometheus endpoint. Subsequently, we employ filtering techniques, including commands like grep and awk, to extract the available space and total capacity values. Then, we proceed to convert these values into gigabytes (GB). Lastly, we present these converted values on the terminal, offering insights into the PVC storage utilization and capacity.

4. Conclusion

In this article, we’ve explored various methods to effectively monitor the remaining space in a Persistent Volume Claim (PVC) within Kubernetes clusters. By leveraging command-line tools such as kubectl, we can inspect the PVC and their associated Pods, enabling us to determine the utilized space and calculate the available storage.

Additionally, we’ve demonstrated how Prometheus metrics can provide real-time insights into PVC capacity and utilization, offering a comprehensive approach to managing storage resources in Kubernetes environments.