View containerd Logs

1. Overview

Whenever we have issues running our workloads in containerd, we often check the logs to see where things went wrong. However, viewing logs in containerd can be a bit tricky because it’s not supported out-of-the-box.

In this tutorial, we’ll see how to access logs for containerd and container workloads using popular clients like nerdctl and crictl.

2. Where Does containerd Store Logs?

containerd, as its name suggests, is meant to be run as a daemon in Linux. This means it can run either as a systemd service (which manages the daemon) or directly in the terminal. In either case, the logs can be accessed:

• As a systemd service – systemd’s journald stores the logs related to the containerd service.
• Using the containerd command – we can see its logs in the standard output of the terminal.

When it comes to container workloads, containerd doesn’t provide a mechanism to store their logs. For that, we have to use a client which manages them for us.

3. Accessing containerd Logs

We can access containerd logs via journalctl when containerd is running as a systemd service, or directly in the terminal’s standard output if we’re executing the containerd command manually in the terminal.

3.1. Using journalctl

In most modern systems, containerd runs as a systemd service. This means that we can easily view and work with the logs collected by journald using the journalctl command.

Let’s view the latest logs using journalctl:

# journalctl -xeu containerd.service

Similarly, we can view logs generated in the last 40 minutes:

# journalctl -xeu containerd --since "40 min ago"

However, these commands only work if containerd is already running as a systemd service. Otherwise, we’ll need to set up containerd as a systemd service beforehand.

3.2. Using the containerd Command

If we’re running containerd directly in the terminal, we can access its logs through the standard output. This can be useful for debugging or monitoring the daemon’s activity in real-time.

Let’s start containerd and view its logs in real-time:

# containerd

This will start the containerd daemon, and we’ll see log messages appearing in the terminal as they’re generated. Consequently, these logs can be used to monitor errors, warnings, or informational messages.

Even better, we can run containerd in the background and direct its logs to containerd.log:

# containerd > containerd.log 2>&1 &

We can then view the logs later using a text editor or command-line tools like tail or less.

4. Accessing Logs for Each Container

containerd doesn’t provide a way to access logs for each container. Instead, the client we use to manage the containers encapsulates this logic. The default client (ctr) doesn’t provide us with a way to view the container logs.

Let’s look at how we can view logs in some of the most popular clients.

4.1. Using nerdctl

contaiNERD CTL (nerdctl) is an advanced client for containerd which provides a Docker-like interface. Let’s install nerdctl from its official release page, and view logs using a Docker-like syntax:

# nerdctl logs <container_name_or_id>

Similarly, to follow along the logs:

# nerdctl logs -f <container_name_or_id>

nerdctl stores the logs generated by the containers in the /var/lib/nerdctl/ directory. The most important thing to remember is that nerdctl only supports viewing logs correctly for containers running with the -d (daemon) flag.

4.2. Using crictl

containerd has a built-in CRI-compatible API, which is why we can also use crictl to interact with it. Now, let’s install crictl using its installation guide and then view the logs for our container:

# crictl logs <container_name_or_id>

To follow along the logs, we can use the -f or –follow flag:

# crictl logs -f <container_name_or_id>

Compared to nerdctl, crictl supports several stable flags that work consistently well with containerd.

4.3. Using /var/log/containers Directory

Usually, we use containerd as a container engine in a Kubernetes cluster. In such a case, kubelet (the Kubernetes agent on each node) stores the logs related to each container running on the node in the /var/log/containers directory.

The path for the file containing the logs of a container follows a specific syntax:

/var/log/containers/<pod_name>_<namespace>_<container_name>-<container_id>.log

Let’s view the logs for a container named ecommerce-backend in the backend pod of the public namespace:

$ cat /var/log/containers/backend_public_ecommerce-backend-545wt6s8fd8s7df8sd7af68.log

In this case, the string after the container name is a randomly generated string that uniquely identifies the container in a deployment.

5. Conclusion

In this article, we discussed the types of logs in containerd, specifically logs generated by containerd itself and logs generated by the containers. journald collects the logs generated by the containerd service and provides various ways to view the logs.

Finally, we discussed the collection of logs generated by containers, which largely depends on the client running those containers. In particular, we saw some popular clients for containerd, like nerdctl and crictl, which provide built-in functionality to view these logs.