1. Overview

When working with Linux servers, we may encounter the “Too many open files” error. In this article, we’ll go over what this error means and how we can fix it.

2. What Is a File Descriptor?

From regular data to network sockets, everything is a file in Linux! A file descriptor is a non-negative integer identifier for an open file in Linux. Each process has a table of open file descriptors where a new entry is appended upon opening a new file.

For example, what happens when we cat a file? It opens the file passed as an argument via the open() system call and gets assigned a file descriptor for it. Then, it interacts with the file through the file descriptor – in this case, just for showing its contents – and finally, closes it via the close() system call.

A process has three file descriptors open by default, denoted by 0 for stdin, 1 for stdout, and 2 for stderr.

3. Checking Open File Descriptors

We can check the number of file descriptors used by various agents to determine where our resources are being used.

3.1. Global Usage

If we want to check the total number of file descriptors open on the system, we can use an awk one-liner to find this in the first field of the /proc/sys/fs/file-nr file:

$ awk '{print $1}' /proc/sys/fs/file-nr
2944

3.2. Per-Process Usage

We can use the lsof command to check the file descriptor usage of a process. Let’s take the caddy web server as an example:

$ sudo lsof -p $(pidof caddy)
COMMAND PID  USER   FD      TYPE  DEVICE SIZE/OFF    NODE NAME
caddy   135 caddy  cwd       DIR   254,1     4096 1023029 /etc/sv/caddy
caddy   135 caddy  rtd       DIR   254,1     4096       2 /
caddy   135 caddy  txt       REG   254,1 33595392 1542819 /usr/bin/caddy
caddy   135 caddy    0u      CHR     5,1      0t0      12 /dev/console
caddy   135 caddy    1u      CHR     5,1      0t0      12 /dev/console
caddy   135 caddy    2w     FIFO     0,9      0t0     119 pipe
caddy   135 caddy    3u  a_inode    0,10        0       6 [eventpoll:2,4,6,7,8,9,10,11,13,15,16,17,18,19,20,21,22,23,24,25,27,28,29,30,31,32,33,34]
caddy   135 caddy    4r     FIFO     0,9      0t0     171 pipe
caddy   135 caddy    5w     FIFO     0,9      0t0     171 pipe
caddy   135 caddy    6u     IPv4     234      0t0     TCP localhost.localdomain:2019 (LISTEN)
caddy   135 caddy    7u     IPv6     240      0t0     TCP *:443 (LISTEN)
caddy   135 caddy    8u     IPv6     241      0t0     TCP *:80 (LISTEN)
caddy   135 caddy    9u     IPv6 3819034      0t0     TCP 66.46.134.82.baeldung.com:443->google.com:53222 (ESTABLISHED)

We can see the exact file that a given file descriptor belongs to under the NAME column, while its type is located under the TYPE column. Looking at the entries with IPv6 as their type, we can conclude that even network sockets take up file descriptors, just like regular files.

4. File Descriptor Limits

A process has certain limits with regard to the number of file descriptors it can have open at a time. One is a soft limit, which can be changed by any unprivileged user and can never exceed the hard limit. An unprivileged user can lower the hard limit but not raise it again, whereas a privileged user such as root can raise and lower it as required.

4.1. Per-Session Limit

We use the ulimit command with the -Sn flag to check the soft limit, and the -Hn flag to check the hard limit for the current session:

$ ulimit -Sn
1024
$ ulimit -Hn
4096

4.2. Per-Process Limit

We can check a process’s limits, given its PID, via the procfs filesystem:

$ pid=31540
$ grep "Max open files" /proc/$pid/limits
Max open files            1024                 4096                 files

The second and third fields correspond to the soft and hard limits, respectively.

4.3. Global Limit

There’s a system-wide limit to the total number of file descriptors that can be open by all the processes combined. This limit is stored in the /proc/sys/fs/file-max file:

$ cat /proc/sys/fs/file-max
1634185

5. Increasing File Descriptor Limits

Now that we have a good understanding of the idea behind the “Too many open files” error, let’s go over various ways to solve it. We can verify these changes with the commands mentioned in the previous section. We use 500000 to refer to the desired limit under this section’s examples.

5.1. Temporarily (Per-Session)

Let’s try to configure the limit for our current session with the ulimit -n command:

$ ulimit -n 4096
$ ulimit -n
4096
$ ulimit -n 8192
sh: error setting limit: Operation not permitted

We cannot set the limit above 4096 as that is the hard limit in this case. And as noted above, only a privileged user can change the hard limit.

5.2. Per-User

We can change the soft and hard limits globally for all processes by appending a few lines to the /etc/security/limits.conf file and re-logging in:

*         hard    nofile      500000
*         soft    nofile      500000
root      hard    nofile      500000
root      soft    nofile      500000

In the first field, we specify the user that the limit affects. We set the limit for all users on the system with the * glob.

Additionally, we might need to append a line to /etc/pam.d/common-session on some systems:

session required pam_limits.so

5.3. Per-Service

On systemd-based distributions, we use the systemctl command to configure the limits for specific services. Taking apache as an example, let’s create a filelimit.conf file for it with tee:

$ sudo mkdir -p /etc/systemd/system/apache.service.d/
$ sudo tee /etc/systemd/system/apache.service.d/filelimit.conf <<EOF
[Service]
LimitNOFILE=500000
EOF

Now, we simply reload the configuration and restart the service:

$ sudo systemctl daemon-reload
$ sudo systemctl restart apache.service

5.4. Globally

Earlier, we checked the system-wide aggregate limit in the /proc/sys/fs/file-max file. We can configure it with sysctl by appending a line to the /etc/sysctl.conf file:

fs.file-max = 500000

Finally, we need to run the sysctl -p command to reload the configuration file.

6. Conclusion

In this article, we learned about the working of Linux File Descriptors, including various ways of checking and configuring their limits depending on our needs. Along the way, we saw how to check the maximum number of open file descriptors globally, per-session, and per-process, as well as ways to configure the limits at the user, session, process, and service levels.

Comments are closed on this article!