How to List Recently Deleted Pods in Kubernetes

1. Introduction

In this tutorial, we’ll delve into the methods for listing recently deleted pods in Kubernetes. First, we’ll review how the lifecycle of a pod works. Then, we’ll understand some commands for listing recently deleted pods. Finally, we’ll see an example of how to run these commands in a Kubernetes environment.

2. Exploring Kubernetes Pod Lifecycle

Before learning how to list deleted pods, it’s essential to understand the lifecycle of a Kubernetes pod. A pod is the smallest and simplest unit that can be created and managed in Kubernetes. It encapsulates one or more closely associated containers, storage resources, a unique network IP, and settings defining how the containers should operate. Thus, a pod represents a singular instance of an application.

For better management, pods go through various phases in their lifecycle. Each one gives us an overview of their condition at that time. The phases are:

It’s important to note that pods are only programmed once during their lifetime. In other words, once created on the intended node, it remains until finished or deleted. If any changes or updates are necessary for the pod, the controller will create new pods based on the updated model to replace the existing ones.

Similarly, the system never relocates a pod to a different node. If necessary, the system replaces the original pod with a new one that’s almost identical, even retaining the same name if it’s important, but its UID will be different.

3. Listing Recently Deleted Pods

Given the characteristics of a pod’s lifecycle, it’s not possible to directly access information about deleted pods using standard commands. However, we can take an approach that uses the Kubernetes event system.

Kubernetes generates events to signal various changes in the pod lifecycle. With this, we can check some important information about recently deleted pods, such as their namespaces and deletion timestamps.

With this approach, we’ll get a list of recently deleted pods. However, this list will only include pods that the system deleted within the last hour. This is the default TTL for events in Kubernetes, so this is also the time they’re stored.

To get a list with only the names of the deleted pods, we can run the following command:

$ kubectl get event -o custom-columns=NAME:.metadata.name | cut -d "." -f1

In it, we’re formatting the output of the events using -o custom-columns=NAME:.metadata.name. In other words, we’re creating a column called NAME and extracting the value of the deleted pod’s metadata.name field to fill that column.

To obtain a list of common information about the pod’s event, we can use the command:

$ kubectl get events --field-selector reason=Killing --sort-by='.metadata.creationTimestamp'

Using the –field-selector reason=Killing argument, we filter the events to return. So, we chose the reason Killing, which indicates that we’re interested in events related to the finalization of pods.

For a better view, we sort the returned list based on the metadata creation timestamp. To do this, we use the argument –sort-by=’.metadata.creationTimestamp. This means listing the pods chronologically, from oldest to newest.

These two commands provide basic information about the deleted pod. Note that if the pod has been deleted for more than 60 minutes, these commands won’t return the requested information.

4. Example

Let’s now run the commands presented in a Kubernetes environment and analyze the output. Then, we need to have a pod created and recently deleted.

4.1. Creating a Pod

First, we need to create a pod:

$ kubectl run my-pod --image=nginx

In this command, we create a pod named my-pod to run an Nginx container. This follows the imperative creation format, where a command directly triggers an action in Kubernetes.

The other creation approach is declarative, where a configuration file (YAML or JSON), also called a manifest, defines the properties of the pod. After interpreting the file, Kubernetes creates the pod based on the specified settings.

Since our focus is only on listing a recently deleted pod, we won’t go into other specific details.

4.2. List Pods

To verify that the system has created our pod, we can list all pods in the cluster using the following command:

$ kubectl get pods
NAME     READY   STATUS    RESTARTS   AGE
my-pod   1/1     Running   0          13s
nginx    1/1     Running   0          10h

As we can see, our pod is running successfully.

4.3. Deleting a Pod

Now, let’s delete our pod:

$ kubectl delete pod my-pod

To check if the system has deleted our pod, we can list all the pods again. This will show that it’s not on the list of active pods:

$ kubectl get pods
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          10h

4.4. Listing Deleted Pods

As explained earlier, one of the ways to list deleted pods is by filtering only their name (or the selected metadata):

$ kubectl get event -o custom-columns=NAME:.metadata.name | cut -d "." -f1
NAME
my-pod
my-pod
my-pod
my-pod
my-pod
my-pod

In this example, the deleted pod generated 6 events. While we’re filtering, we only have access to the pod’s name and no other information.

To view more information, we can run:

$ kubectl get events --field-selector reason=Killing --sort-by='.metadata.creationTimestamp'
LAST SEEN   TYPE     REASON      OBJECT       MESSAGE
20m         Normal   Scheduled   pod/my-pod   Successfully assigned default/my-pod to kubernetes-n2
20m         Normal   Pulling     pod/my-pod   Pulling image "nginx"
19m         Normal   Pulled      pod/my-pod   Successfully pulled image "nginx" in 8.108s (8.108s including waiting)
19m         Normal   Created     pod/my-pod   Created container my-pod
19m         Normal   Started     pod/my-pod   Started container my-pod
17m         Normal   Killing     pod/my-pod   Stopping container my-pod

With this command, we can trace events from pod creation to termination. In this simple scenario, the system displays only normal events. A broader range of events and types may appear in complex environments (such as production settings).

5. Conclusion

In this article, we’ve studied the process of listing deleted pods in Kubernetes. As we’ve learned, we need to use the Kubernetes event system to find out about these pods. With this, we can list and analyze deletion events.