inotify and Solving the “upper limit on inotify watches reached!” Error

1. Introduction

Modern software often updates views automatically based on data changes:

• head and tail
• git (Global Information Tracker), to follow file changes for commits
• vscode (Visual Studio Code)

This convenience is afforded by different systems.

In this tutorial, we talk about the inotify system and how it functions. After that, we explore ways of checking its current users. Finally, we discuss some common errors linked with inotify.

We tested the code in this tutorial on Debian 11 (Bullseye) with GNU Bash 5.1.4. It is POSIX-compliant and should work in any such environment.

2. Filesystem Event Notifications

We’ve discussed inodes as a key in native Linux filesystems. In fact, their change is linked with many file events.

Based on this, the inotify subsystem’s main functions are to monitor and report on events in the filesystem. Whether with the inotifywait and inotifywatch tools, or system calls from other software, we can easily keep track of file operations.

For example, we can use a command line to simply wait for any event on a given file:

$ inotifywait /file.ext && echo 'Event.'
Setting up watches.
Watches established.

Consequently, if /file.ext is now part of any operation, we’ll get a notification. But what if we wanted to permanently do that for many objects at once?

3. Monitoring Multiple Files and Directory Trees

There are times when we need more general monitoring. One way is to add the –recursive (-r) and –monitor (-m) switches to inotifywait. They ensure directory tree processing and constant monitoring, respectively.

Additionally, we can use inotifywatch to listen for events linked with a given part of the filesystem:

$ inotifywatch /
Establishing watches...
Finished establishing watches, now collecting statistics.

Since a separate watch is needed for each filesystem object, Linux has an upper limit on its maximum number. In fact, we can check what it is in a given system:

$ cat /proc/sys/fs/inotify/max_user_watches
524288

Here, the system allows 524288 (2^19) watches. Conveniently, we can see who uses them at any moment.

4. Checking inotify Users

By combining several Linux commands, we can find out the processes that currently use watches:

$ inotifywatch / &
Establishing watches...
Finished establishing watches, now collecting statistics.
[1] 666
$ find /proc/*/fd -lname anon_inode:inotify |
  cut -d/ -f3 |
  xargs -I '{}' -- ps --no-headers -o '%p %U %c' -p '{}' |
  uniq -c |
  sort -nr
      1    6660 root     inotifywatch

The number of inotify file descriptors is not the same as the number of watches. Because of this, there is a script, which contains a better version of the above command line along with other features.

Additionally, we can use strace (System Trace) to get a list of all watches:

$ strace --follow-forks --trace='inotify_add_watch' inotifywait --quiet /file.ext
inotify_add_watch(3, "/file.ext",
  IN_ACCESS|IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_CLOSE_NOWRITE|IN_OPEN|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF) = 1

Finally, there is also the lsof (List Open Files) tool, which also lists inotify users:

$ tail -f /file.ext &
[1] 666
$ lsof | grep inotify
tail    666             root    4r  a_inode               0,11        0            15387 inotify

Even with all the ways to use and handle watches, issues are not rare.

5. Common inotify Errors

There are pitfalls when using the inotify subsystem. Let’s discuss some of them.

5.1. Permissions

Indeed, as with other filesystem operations, we need to have at least some control over the target objects. In other words, we can’t monitor without proper rights:

$ inotifywatch /root
Establishing watches...
Failed to watch /root: Permission denied

To fix this, we need to be able to at least read the target object. Here, we add the necessary right via chmod (Change Mode):

$ chmod +r /root

After these changes, we can add watches.

5.2. inotify Watch Limit

Consider that a recursive inotifywatch on / would take up as many watches as there are files. Indeed, doing that on a filesystem with more files than the watch limit would produce an error:

$ cat /proc/sys/fs/inotify/max_user_watches
524288<
$ df --inodes /
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
/dev/sdb       16777216 666000 16111216    4% /
$ inotifywatch --recursive /
Establishing watches...
Failed to watch /; upper limit on inotify watches reached!
Please increase the amount of inotify watches allowed per user via '/proc/sys/fs/inotify/max_user_watches'.

Of course, the solution is in the error text. However, there is more than one way to apply it:

$ max_user_watches=CUSTOM_MAX_USER_WATCHES_VALUE
$ sysctl fs.inotify.max_user_watches=${max_user_watches}
$ echo fs.inotify.max_user_watches=${max_user_watches} >> /etc/sysctl.conf
$ sysctl --load

While the first command applies it per session, the second survives a reboot. That’s because it adds a line to /etc/sysctl.conf. However, if the line is already there, we should change it per our needs. We can avoid a restart with the last command, which rereads and applies the configuration file.

5.3. Main Memory (RAM) Issues

Because each watch is a structure, available memory is also a bottleneck for using inotify. In fact, a watch can take up to 1KB of space. This means a million watches could result in 1GB of extra RAM usage.

Sometimes, this can lead to strange errors:

• inotify_add_watch failed: No space left on device (/file.ext)
• cannot watch ‘/file.ext’: No space left on device

Freeing up memory seems like the way to go. However, there are a couple of problems with this approach.

Firstly, inotify uses kernel memory, which can’t be swapped. Secondly, programs can throw such errors even when just running out of watches.

6. Summary

In this article, we discussed the inotify Linux subsystem, used by many applications.

In conclusion, knowing about watches allows us to consider not only the inner workings of software products but also understand errors we see during their use.