Kubernetes Pod and Container Restarting

Last updated: March 26, 2024

1. Introduction

Kubernetes is a platform for managing container deployments in an organized fashion. Further, it has features that provide redundancy and control over parts of a deployment. However, the smallest officially manageable unit in Kubernetes is a pod. Although a pod can contain a single container, that’s not always the case.

In this tutorial, we talk about pod and container restarting within a Kubernetes cluster. First, we explore a simple Kubernetes pod definition, and how to deploy and inspect it. After that, we go through ways to restart a pod within a Deployment. Next, we see how to restart an isolated pod. Finally, we look at three ways to restart and manage a single container within a pod.

We tested the code in this tutorial on Debian 12 (Bookworm) with GNU Bash 5.2.15. Unless otherwise specified, it should work in most POSIX-compliant environments.

2. Sample Kubernetes Pod

Since we aim to control sub-pod objects, let’s start by understanding how to create a single atomic unit of Kubernetes.

In particular, we create a pod with two containers:

$ printf '
apiVersion: v1
kind: Pod
metadata:
  name: compod
spec:
  containers:
  - name: deb1
    image: debian:latest
    command: ["bash"]
    tty: true
  - name: deb2
    image: debian:latest
    command: ["bash"]
    tty: true
' | kubectl apply --filename=-
pod/compod created

Here, we pipe the pod definition in YAML format from printf. The kubectl tool processes that through stdin as indicated by the – dash after –filename.

Specifically, the two containers are [name]d deb1 and deb2 respectively. Both use the debian:latest Debian image and run bash as their main command. Notably, we use tty to preserve the running state and the ability to enter each container.

Let’s get the basic listing of the resulting pod:

$ kubectl get pod/compod
NAME     READY   STATUS    RESTARTS   AGE
compod   2/2     Running   0          1m

Notably, we don’t need to specify a –namespace since we just use the default.

Next, we can check the list of containers that comprise the pod:

$ kubectl get pod/compod --output=jsonpath='{.spec.containers[*].name}'
deb1 deb2

In this case, –output extracts a specific jsonpath that contains the names of each container. As expected, the output includes both names.

3. Restart Pod Within Deployment

There are different ways to restart a Kubernetes pod within a Deployment. Some are direct, while others are more involved.

3.1. ReplicaSet Pod

A ReplicaSet is a mechanism that ensures redundancy by keeping a certain number of pod duplicates running. For instance, if a pod has a replicas fields set to 6, Kubernetes aims to keep exactly that number up at any given time.

Let’s create a Deployment with 3 replica sets:

$ printf '
apiVersion: apps/v1
kind: Deployment
metadata:
  name: comdep
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ox
  template:
    metadata:
      labels:
        app: ox
    spec:
      containers:
      - name: deb1
        image: debian:latest
        command: ["bash"]
        tty: true
      - name: deb2
        image: debian:latest
        command: ["bash"]
        tty: true
' | kubectl apply --filename=-
deployment.apps/comdep created

Here, we can see the original pod definition within the spec.template.spec path of the Deployment. In addition, the definition contains the replicas field, a selector, and the app=ox label.

The label is a way to refer to all pods within the Deployment:

$ kubectl get pods --selector=app=ox
NAME                      READY   STATUS    RESTARTS   AGE
comdep-7df9d4d57d-qs5p6   2/2     Running   0          57s
comdep-7df9d4d57d-qz7j9   2/2     Running   0          57s
comdep-7df9d4d57d-t4jml   2/2     Running   0          57s

Conversely, we can use the delete subcommand and the app=ox label via –selector:

$ kubectl delete pods --selector=app=ox
pod "comdep-7df9d4d57d-qs5p6" deleted
pod "comdep-7df9d4d57d-qz7j9" deleted
pod "comdep-7df9d4d57d-t4jml" deleted

This affects the way we restart a pod because the controller attempts to recreate any deleted pod:

$ kubectl get pods --selector=app=ox
NAME                      READY   STATUS              RESTARTS   AGE
comdep-7df9d4d57d-2lcfb   0/2     ContainerCreating   0          4s
comdep-7df9d4d57d-hwdmp   0/2     ContainerCreating   0          4s
comdep-7df9d4d57d-qs5p6   2/2     Terminating         0          9m34s
comdep-7df9d4d57d-qz7j9   2/2     Terminating         0          9m34s
comdep-7df9d4d57d-t2frd   0/2     ContainerCreating   0          4s
comdep-7df9d4d57d-t4jml   2/2     Terminating         0          9m34s

Thus, we restart each pod in the deployment by just deleting it as long as there is a ReplicaSet backing.

3.2. Rollout Restart

Another way to restart a Deployment pod is to perform a rollout of the whole deployment:

$ kubectl rollout restart deployment comdep
deployment.apps/comdep restarted

While this is a simple method, it can affect pods we don’t want to touch.

4. Restart Isolated Pod

Sometimes, we might just want to restart a single pod, regardless of its participation in a Deployment.

In that case, we can preserve the pod definition and use that to replace the whole instance:

$ kubectl get pod/compod --output=YAML | kubectl replace --force --filename=-
pod "compod" deleted
pod/compod replaced

In summary, we get the definition of pod/compod in the YAML format and pipe the –output to kubectl, which [–force]s a [replace]ment with the same definition.

Notably, we need –force to terminate the existing pod.

5. Restart Container Within Pod

Of course, if a pod is composed of a single container, we don’t need to go beyond a pod restart to have that single container restart as well. However, Kubernetes doesn’t offer a direct method to control separate containers. This means that we might affect containers within a pod that aren’t the target of our operation.

5.1. With Tools Within Container

To avoid restarting containers that we want to preserve, we can employ the exec subcommand as usual and target a specific –container and –pod:

$ kubectl exec --tty --stdin --pod=pod/compod --container=deb1 -- killall5

Let’s break down this command:

  • –pod (-p): name of pod
  • –container (-c): name of container (first is default)
  • –stdin (-i): connect stdin to container
  • –tty (-t): use TTY as stdin
  • killall5: kill all processes

The mechanics behind this approach are similar to the way we access a container via a shell like Bash, for example.

5.2. With Interactive Shell

Of course, we can use an interactive shell to perform any operation on a container, including a reboot:

$ kubectl exec --tty --stdin --pod=pod/compod --container=deb1 -- sh

In this case, we run sh within the deb1 container in the compod pod. The advantage of this approach is that we can perform maintenance and debugging operations instead of a basic restart. On the other hand, not all containers have shells and manual operations can be more tedious and dangerous.

5.3. With Tools From Infused Image

As already mentioned, a container doesn’t always have a shell or killall5 command to work with. In such cases, we can infuse a given container with an –image:

$ kubectl debug --tty --stdin --pod=pod/compod --image=debian:latest --target=deb1 -- killall5

In short, we use the debug command to create a special container in the same pod as the one we want to restart. Further, this new container is temporary, only serves its role of executing the supplied command, and provides the tools in the image it was created from. Moreover, a debug container attempts to access the same resources as the original.

6. Summary

In this article, we talked about targeted and non-targeted pod and container restarting within a Kubernetes environment.

In conclusion, although we can use a Deployment to restart pods, selecting a specific pod or container for an operation provides more control over cluster management.

Comments are open for 30 days after publishing a post. For any issues past this date, use the Contact form on the site.