Get a Continuous Stream of Logs for Pods in Kubernetes

1. Overview

Log streaming in Kubernetes is crucial to monitoring and troubleshooting applications running in containerized environments. It provides real-time access to the logs generated by containers within Pods.

In this article, we’ll learn how to use the kubectl logs command to get a continuous stream of logs from the Kubernetes pods.

2. Understanding the Scenario

Let’s start with the ubuntu-pod.yaml configuration file for a pod that uses the ubuntu image and prints a message every minute:

$ cat ubuntu-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-pod
spec:
  containers:
  - name: ubuntu-container
    image: ubuntu
    command: ["bash", "-c", "while true; do echo 'running ...' && sleep 60 done"]

We can see an infinite while loop specified in the command attribute. So, the pod should stay up and running forever while logging a message every minute.

Now, let’s use the kubectl apply command to create the ubuntu-pod from our configuration file:

$ kubectl apply -f ubuntu-pod.yaml
pod/ubuntu-pod created

Further, we can verify that the pod is indeed up and running:

$ kubectl get pods --field-selector metadata.name=ubuntu-pod
NAME         READY   STATUS    RESTARTS   AGE
ubuntu-pod   1/1     Running   0          3m7s

Lastly, let’s use the kubectl logs command to check the logs from our ubuntu-pod:

$ kubectl logs ubuntu-pod
running ...
running ...
running ...
running ...
running ...
$ 

Great! We can see the log messages generated so far on the console. After that, the control is transferred to the command prompt ($), and we no longer see a stream of logs. However, if we were troubleshooting an issue, a live stream of logs could reflect the latest state of the application and help with effective monitoring.

So, let’s get started and learn the techniques we can use to get a stream of logs from our pod.

3. Streaming Logs Using the –follow Option

Let’s use the –follow option with the kubectl logs command to see a stream of logs from the ubuntu-pod:

$ kubectl logs --follow ubuntu-pod
running ...
running ...
running ...
running ...
running ...
running ...
running ...

Interestingly, this approach doesn’t transfer the control to the command prompt ($). So, let’s wait for a minute to see the next message pop up on our console:

running ...

Fantastic! As expected, a new log message appeared. That’s because our pod is generating the log message every minute.

Nonetheless, there is much more to learn about streaming logs in Kubernetes. So, let’s get ready to dive deeper into this.

4. Enriching the Log Stream

In this section, we’ll learn how to enrich our log stream messages with metadata using the options available with the kubectl logs command.

4.1. With Timestamps

Knowing the timestamp of a log message is perhaps the most important detail besides the message itself. The kubectl logs command provides the –timestamps option that we can use to append the timestamps at the beginning of the message.

Let’s see this in action for the ubuntu-pod that’s running in our Kubernetes cluster:

$ kubectl logs --follow --timestamps ubuntu-pod
2023-07-30T03:09:49.368021900Z running ...
2023-07-30T03:10:49.368683774Z running ...
2023-07-30T03:11:49.370931509Z running ...

We can notice that messages are showing up along with timestamp detail. Furthermore, we must remember that the timestamps are in the RFC3339 format.

Now, whenever a new message pops up in our stream, it’s easily noticeable because of the guaranteed change in the logline:

2023-07-30T03:12:49.372825176Z running ...

Wonderful! Our log stream looks more useful now.

4.2. With Source

Another useful detail to add to our log stream is its source. Fortunately, the kubectl logs command has us covered here as well with its –prefix option.

Let’s move forward and use the –prefix option to enrich our stream with the source of log messages:

$ kubectl logs --follow --prefix ubuntu-pod
[pod/ubuntu-pod/ubuntu-container] running ...
[pod/ubuntu-pod/ubuntu-container] running ...

As expected, we can notice that our stream shows the pod name (ubuntu-pod) and container name (ubuntu-container) for each logline. With this information, we can trace each logline to its correct source.

5. Limiting Log Stream by Size

In our scenario, we’ve added quite a simple log message with enough delay between any two log messages. However, for an application solving a real business need, the application could be logging longer messages at a significantly higher frequency. So, let’s learn how to limit the streaming by the size of logs.

5.1. With the –tail Option

When we use the –follow option to get the log stream from a running pod, Kubernetes first shows the historical log messages. Once all messages are shown, we can see the near real-time live stream of logs. However, such behavior could add noise if the pod ran for many days or generated a high volume of log events in a short period.

For such scenarios, we can use the –tail option to limit the size of historical loglines to a fixed number. As a result, we should see our log stream sooner.

Let’s verify our understanding with the log stream from the ubuntu-pod pod:

$ date
Sun Jul 30 03:19:14 AM UTC 2023
$ kubectl logs --follow --timestamps --tail 2 ubuntu-pod
2023-07-30T03:17:49.383687361Z running ...
2023-07-30T03:18:49.386528064Z running ...

It works as expected. In the output, we can see that only two loglines from the past showed up.

5.2. With the –limit-bytes Option

Log size is another criterion we can consider to restrict the size of our log stream. For this purpose, we can use the –limit-bytes option with the kubectl logs command.

Let’s start by checking out how many characters per byte are shown in our log stream by passing a value of 1 byte to the –limit-bytes option:

$ kubectl logs --follow --limit-bytes 1 ubuntu-pod
r
$

We can see that one byte corresponds to one character in this case.

Since, in our scenario, each logline has 11 characters, it’s worth verifying this by passing 11 bytes to the –limit-bytes option:

$ kubectl logs --follow --limit-bytes 11 ubuntu-pod
running ...
$

Perfect! We’ve validated our understanding. However, we must notice that the control was transferred to the command prompt ($) immediately in both cases. That’s because the –limit-bytes option applies to the entire log stream, not just the historical logs.

Lastly, let’s pass 500 bytes with the –limit-bytes option to see the log stream from ubuntu-pod:

$ kubectl logs --follow --limit-bytes 500 ubuntu-pod
running ...
running ...
running ...
running ...
running ...
running ...
running ...
running ...
running ...
running ...
running ...
running ...
running ...
running ...

In this case, it’s worth noting that the stream will close as soon as it gets 500 characters.

6. Limiting Log Stream by Time

In this section, let’s see how to limit the log stream using time-based options, such as –since and –since-time.

6.1. With the –since Option

We can specify a time duration with the –since option to restrict the historical log lines to a specific time in the past.

Let’s start by passing 300s (300 seconds) to the –since option and see the log stream from ubuntu-pod:

$ date
Sun Jul 30 03:28:51 AM UTC 2023
$ kubectl logs --follow --timestamps --since 300s ubuntu-pod
2023-07-30T03:24:49.398533266Z running ...
2023-07-30T03:25:49.401017157Z running ...
2023-07-30T03:26:49.402346354Z running ...
2023-07-30T03:27:49.404715543Z running ...
2023-07-30T03:28:49.407544746Z running ...

We can notice that only five log lines from the past are showing up in the log stream. That’s because our application logs once per minute, which calculates to 5 times in 300 seconds.

Furthermore, we can also pass the duration in minutes. Let’s verify this by passing 2m with the –since option:

$ date
Sun Jul 30 03:29:53 AM UTC 2023
$ kubectl logs --follow --timestamps --since 2m ubuntu-pod
2023-07-30T03:28:49.407544746Z running ...
2023-07-30T03:29:49.409491441Z running ...

The result looks correct.

Next, let’s pass 0.05h (3 minutes) with the –since option to confirm that we can even pass fractional arguments:

$ date
Sun Jul 30 03:31:43 AM UTC 2023
$ kubectl logs --follow --timestamps --since 0.05h ubuntu-pod
2023-07-30T03:28:49.407544746Z running ...
2023-07-30T03:29:49.409491441Z running ...
2023-07-30T03:30:49.411283477Z running ...

Lastly, let’s see an interesting scenario where we pass 2m120s with the –since option:

$ date
Sun Jul 30 03:30:36 AM UTC 2023
$ kubectl logs --follow --timestamps --since 2m120s ubuntu-pod
2023-07-30T03:26:49.402346354Z running ...
2023-07-30T03:27:49.404715543Z running ...
2023-07-30T03:28:49.407544746Z running ...
2023-07-30T03:29:49.409491441Z running ...

We can notice that four loglines show up in the stream. It helps us understand that the total duration is computed by adding all the individual durations, and nothing gets excluded based on priority for time units.

6.2. With the –since-time Option

If we want to restrict the log stream to only show messages after a specific timestamp, we can use the-since-time option.

Let’s check out the current time and pass a timestamp from a few minutes in the past with the –since-time option:

$ date
Sun Jul 30 03:35:47 AM UTC 2023
$ kubectl logs --follow --timestamps --since-time 2023-07-30T03:30:00.000000000Z ubuntu-pod
2023-07-30T03:30:49.411283477Z running ...
2023-07-30T03:31:49.414355691Z running ...
2023-07-30T03:32:49.416516375Z running ...
2023-07-30T03:33:49.418557475Z running ...
2023-07-30T03:34:49.420578582Z running ...

It works just fine. Since we passed a timestamp, 5 minutes into the past, we see only five historical loglines as the application logs one message per minute.

Furthermore, if we pass a timestamp that is in the future, our log stream will not show anything until the time is right:

$ kubectl logs --follow --timestamps --since-time 2024-07-30T03:30:00.000000000Z ubuntu-pod

So, we can use such behavior if we want to wait for a specific time before getting the log stream.

7. Streaming Logs for Multiple Containers

In this section, we’ll learn how to stream logs from a pod with multiple containers inside.

7.1. Setup

First, let’s write the pod configuration such that it defines multiple containers, namely, ubuntu-container-1 and ubuntu-container-2:

$ cat ubuntu-multi-containers.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-pod-multi-containers
spec:
  containers:
  - name: ubuntu-container-1
    image: ubuntu
    command: ["bash", "-c", "while true; do echo 'running container-1...' && sleep 60; done"]
  - name: ubuntu-container-2
    image: ubuntu
    command: ["bash", "-c", "while true; do echo 'running container-2...' && sleep 60; done"]

Next, let’s apply this configuration to create the pod containing multiple containers:

$ kubectl apply -f ubuntu-multi-containers.yaml 
pod/ubuntu-pod-multi-containers created

Further, we can verify that the pod is up and running and 2/2 containers are READY:

$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
ubuntu-pod-multi-containers   2/2     Running   0          58s

Lastly, let’s see if we can use the existing approach to see logs from the pod:

$ kubectl logs --follow ubuntu-pod-multi-containers
Defaulted container "ubuntu-container-1" out of: ubuntu-container-1, ubuntu-container-2
running container-1...
running container-1...

Although we can see the log stream, it’s from a single container.

7.2. Stream From Specific Container

We can use the –container option to pass the container name. Let’s use this to get the log stream from the ubuntu-container-2 container explicitly:

$ kubectl logs --follow ubuntu-pod-multi-containers --container ubuntu-container-2
running container-2...
running container-2...

It looks like we’ve got this one right.

7.3. All Containers

If we specify multiple container names by passing the –container option multiple times, we get the log stream from the last container in the list. Let’s verify this behavior:

$ kubectl logs -f ubuntu-pod-multi-containers --container ubuntu-container-1 --container ubuntu-container-2
running container-2...
running container-2...

For a scenario where we want to get logs from all the containers running inside the pod, we can pass the –all-containers option:

$ kubectl logs -f ubuntu-pod-multi-containers --all-containers
running container-2...
running container-2...
running container-1...
running container-1...

Fantastic! We’ve nailed this one.

8. Streaming Logs for Deployment Pods

In this section, we’ll learn how to stream logs from pods managed with a Kubernetes Deployment resource.

8.1. Setup

First, let’s look at the ubuntu-deployment.yaml configuration for our deployment resource:

$ cat ubuntu-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ubuntu-app
  template:
    metadata:
      labels:
        app: ubuntu-app
    spec:
      containers:
      - name: ubuntu-container
        image: ubuntu
        command: ["bash", "-c", "while true; do echo $(hostname)' is running ...' && sleep 60; done"]

Now, let’s apply the configuration to create the ubuntu-deployment deployment:

$ kubectl apply -f ubuntu-deployment.yaml
deployment.apps/ubuntu-deployment created

Lastly, let’s verify that the pods are up and running:

$ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
ubuntu-deployment-7ffd5967d-m2h59   1/1     Running   0          25s
ubuntu-deployment-7ffd5967d-trdv6   1/1     Running   0          25s
ubuntu-deployment-7ffd5967d-zpvxg   1/1     Running   0          25s

We can see that the name of the pods has a dynamic suffix. So, although we can stream logs from individual pods by passing their names, it’s a bit inconvenient.

8.2. Streaming Logs by Labels

We’ve defined the app label with the ubuntu-app value in our deployment configuration. So, all the pods managed by ubuntu-deployment have this label.

Now, if we want to stream logs for all pods within the deployment, we can pass a label query using the -l option:

$ kubectl logs -f -l app=ubuntu-app
ubuntu-deployment-7ffd5967d-m2h59 is running ...
ubuntu-deployment-7ffd5967d-trdv6 is running ...
ubuntu-deployment-7ffd5967d-zpvxg is running ...
ubuntu-deployment-7ffd5967d-zpvxg is running ...
ubuntu-deployment-7ffd5967d-trdv6 is running ...
ubuntu-deployment-7ffd5967d-m2h59 is running ...

Great! We can see that the log stream contains loglines from all three pods.

9. Conclusion

In this article, we learned to use the –follow option with the kubectl logs command for getting a continuous stream of logs from pods running in a Kubernetes cluster. Furthermore, we explored multiple other options, such as timestamps, prefix, since, since-time, limit-bytes, and so on, to fine-tune the logs.

Lastly, we learned how to stream logs for a specific container, all containers in a pod, or all pods managed by a deployment.