How to Fix Docker’s No Space Left on Device Error

1. Overview

While working with Docker containers, we often create various objects such as networks, volumes, images, containers, etc. Docker doesn’t clean up these objects automatically, even if they aren’t getting used. This might start to cause disk space problems.

We might get a no space left on the device error. To avoid such a situation, we must clean up unused Docker objects on a regular basis.

In this short tutorial, we’ll discuss some of the ways to clean things up.

2. Setting up an Example

Let’s create a few Docker objects to occupy the maximum disk space. This will allow us to simulate the error.

2.1. Keeping an Unused Image

First, we’ll pull the NGINX and Redis images and verify that they have been pulled successfully:

$ docker pull nginx
$ docker pull redis

$ docker image list
REPOSITORY   TAG       IMAGE ID       CREATED      SIZE
nginx        latest    51086ed63d8c   5 days ago   142MB
redis        latest    f8528f17261c   5 days ago   117MB

Next, we can create a container using the NGINX image:

$ docker container run -it --name web-server-01 -d nginx:latest

Here, we have kept the Redis image unused intentionally.

2.2. Creating a Dangling Volume

Let’s now create a Docker volume:

$ docker volume create unused-volume

We’ll find its mount point location:

$ docker volume inspect -f '{{ .Mountpoint }}' unused-volume
/var/lib/docker/volumes/unused-volume/_data

Finally, we will create a large file inside the /var/lib/docker/volumes/unused-volume/_data directory (or whatever directory is used by the current docker runtime) using the fallocate command:

$ sudo fallocate -l 6.12G /var/lib/docker/volumes/unused-volume/_data/large-file.img

In the next section, we’ll see how this step helps us to simulate the low disk space error.

2.3. Simulating the Low Disk Space Error

Let’s find out the available disk space on our host using the df command:

$ df -h /
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/almalinux-root  8.0G  8.0G  7.0M 100% /

Here, we can see that now the available disk space is only 7 MB. The Avail column in the output indicates this.

We might try to pull a MySQL image of size 127 MB:

$ docker image pull mysql:latest
latest: Pulling from library/mysql
295ca2342728: Downloading [==================================================>]  40.59MB/40.59MB
79af4312a7e0: Download complete 
write /var/lib/docker/tmp/GetImageBlob151502556: no space left on device

However, as we can see, the command fails with the error – write /var/lib/docker/tmp/GetImageBlob151502556: no space left on device.

In the next section, we’ll see how to get rid of this error by cleaning up unused objects.

3. Cleaning Up Unused Objects

We saw that images and volumes take up space on the Docker host, so it’s a good practice to clean them up if they are not getting used.

However, we cannot remove them carelessly because removing the wrong volume can cause permanent data loss. Hence it’s important to identify and remove the unused object only.

In such cases, we can use the prune child command that allows us to clean up unused objects in a safer way.

3.1. Cleaning Up Unused and Dangling Images

Removing unused and dangling images will help us to reclaim disk space. We can achieve this using the image prune child command:

$ docker image prune -a -f
Deleted Images:
untagged: redis:latest
deleted: sha256:fc24995fdc7260d59418714deda07229a703bca3d64f6f19c7ade1e3c0fb2d55

Total reclaimed space: 36.43MB

As we can see, the command deletes the unused Redis image and reclaims disk space of 36.43 MB.

In this example, the -a option removes the dangling as well as unused images. Without this option, the command removes only dangling images. The -f option makes command behavior non-interactive.

3.2. Cleaning Up Unused Volumes

In a similar way, we can use the volume prune child command to remove the unused volumes. Let’s see this in action:

$ docker volume prune -f
Deleted Volumes:
unused-volume

Total reclaimed space: 6.571GB

In this example, the command reclaims the disk space of 6.571 GB by removing the unused volume. This is the same volume that we created while setting up our example.

3.3. Cleaning Up All Unused Objects

So far, we’ve removed individual Docker objects, such as images and volumes, using the prune child command. However, this isn’t the most efficient way.

We can use the system prune child command to remove multiple unused objects in one go. This command removes all unused networks, images, containers, dangling build caches, and optionally, volumes.

Let’s try this out. We’ll start by stopping the web-server-01 container that we created earlier:

$ docker container stop web-server-01

Now, let’s remove the container and its image using the system prune child command:

$ docker system prune -a -f
Deleted Containers:
0cfeb6d3f7bdc8dc2e41d56987ff30b3e4a6c9b555c640a377ef38eb5fd3cdaf

Deleted Images:
untagged: nginx:latest
deleted: sha256:fe7b1e9bf7922fbc22281bcc6b4f5ac8f1a7b4278929880940978c42fc9d0229

Total reclaimed space: 141.7MB

As we can see, the command deletes the stopped container and the image associated with it to reclaim the disk space.

We should note that by default, this command doesn’t remove any volume to prevent data loss. However, we can instruct it to remove the unused volumes using the –volumes option.

4. Changing the Default Storage Location

Docker uses the directory from the host machine to persist its important data, such as container configurations, network settings, images, volumes, etc. In Docker’s terminology, it’s called the Docker Root Directory, and its location varies with the operating system. We can instruct Docker to use a different storage location by changing its root directory.

In a production environment, the host is often configured with multiple disks. In such a case, we can use one of the larger disks as an image store. However, in this tutorial, we’ll configure just another directory as a root directory.

4.1. Finding the Current Storage Location

First, find out the root directory using the info child command:

$ docker info -f '{{ .DockerRootDir }}'
/var/lib/docker

4.2. Changing the Storage Location

Next,  create a new empty directory and configure it as the root directory by editing the /etc/docker/daemon.json file:

$ mkdir -p /tmp/new-docker-root-dir
$ sudo vi /etc/docker/daemon.json

Then we edit the file, so it has a data-root pointing at the new directory. When we’ve saved it:

$ sudo cat /etc/docker/daemon.json 
{ 
   "data-root": "/tmp/new-docker-root-dir"
}

Finally, we must restart the docker service and check the updated root directory using the info child command:

$ sudo systemctl restart docker
$ docker info -f '{{ .DockerRootDir}}'
/tmp/new-docker-root-dir

4.3. Verifying the Updated Storage Location

Now, let’s pull the Docker image and list the contents of the /tmp/new-docker-root-dir directory:

$ docker image pull alpine:latest
latest: Pulling from library/alpine
213ec9aee27d: Pull complete 
Digest: sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad
Status: Downloaded newer image for alpine:latest
docker.io/library/alpine:latest

$ sudo ls -1 /tmp/new-docker-root-dir
buildkit
containers
image
network
overlay2
plugins
runtimes
swarm
tmp
trust
volumes

Here, we can see that Docker created the required sub-directory structure to store images and other data.

5. Expanding the File System

In the previous section, we saw how to fix the low disk space error by changing the Docker Root Directory. However, this method will not work every time. We’ll observe the same error again once the new root directory gets fully utilized.

To overcome such a case, we can expand the host file system by adding more disks. However, this is possible only when the storage allocation is done using a volume manager. In Linux, we can achieve this using a Logical Volume Manager(LVM).

6. Automating Clean-Up of the Unused Objects

Previously, we executed the prune child command manually to clean up the unused objects. However, in some cases doing such manual tasks is time-consuming and error-prone. For example,  when we want to perform this task on multiple hosts. The situation becomes tricky when the number of hosts increases over a period of time.

In such a scenario, we can use cron for automation. We can configure a cron job to perform the clean-up activity on a regular basis.

7. Conclusion

In this article, we saw how to fix Docker’s no space left on the device error.

First, we used the prune child command to clean up the unused objects. Next, we discussed how to fix the error by changing the default storage location. Then, we discussed how to deal with the error by expanding the file system.

Finally, we discussed how to avoid such a scenario by automating the clean-up process.