Understanding Stale File Handles in Linux

1. Overview

In this tutorial, we’ll deal with stale file handles in Linux. They’re a relatively common occurrence when dealing with files, especially shared. To tackle stale file handles, first, we must clarify the general idea of files.

The code in this tutorial has been tested on Debian 10.10 (Buster) with GNU Bash 5.0.3. It is POSIX-compliant and should work in any such environment.

2. Index Node

Index nodes (inode) are filesystem elements, which describe other similar elements. Essentially, an inode must exist for an object like a file or directory to also exist. This article uses the term file for any filesystem object that has an inode associated with it.

We use the stat (statistics) command to show details of a specific file:

user@baeldung:~$ stat filename
File: filename
Size: 583 Blocks: 8 IO Block: 4096 regular file
Device: 810h/2064d Inode: 666 Links: 1
[...]

The file filename references inode number 666 (last row).

But who references the file?

3. File Handle

File handles (file descriptors) are just integers. Processes use them to index a system table of open files (file descriptions). File handles, unlike filesystem elements, do not reside on or get updated with the filesystem. They’re instead used by processes to keep a register of its open files.

We list this register by applying the ls (list) command with the -l (long listing format) flag on the proc (process information) pseudo-filesystem:

user@baeldung:~$ ls -l /proc/1296/fd
total 0
lrwx------ 1 user user 64 Jul 1 10:49 0 -> /dev/pts/1
lrwx------ 1 user user 64 Jul 1 10:49 1 -> /dev/pts/1
lrwx------ 1 user user 64 Jul 1 10:49 2 -> /dev/pts/1
lrwx------ 1 user user 64 Jul 1 10:50 3 -> /home/user/filename

The output shows that file descriptor 3 points to file /home/user/filename. Using stat with the -L (dereference link) flag on that descriptor shows the inode number of the file:

user@baeldung:~$ stat -L /proc/1296/fd/3
File: /proc/1296/fd/3
Size: 583 Blocks: 8 IO Block: 4096 regular file
Device: 810h/2064d Inode: 666 Links: 1
[...]

Files reference inodes. Processes reference files via file handles. So, how would a file handle become stale?

4. Stale File Handles

Files may exist under many names, pointing to the same inode. However, once an inode is not referenced or hardlinked to, it is freed for future use.

At this point, the related file handles continue pointing to a non-existent or non-matching (different inode number) file. They’re now stale file handles. Even if we recreate the file in question, it is nearly impossible that the kernel reuses the same inode number.

The resulting error comes about in different ways, but the standard code is ESTALE 116 with the message Stale file handle:

user@baeldung:~$ cd /mounted
-bash: cd: /mounted: Stale file handle

If the process does not handle the error, it may just not output anything or claim missing files.

This situation generally has one standard fix.

5. Resolve Stale File Handles

Stale file handles are refreshed when the process reopens the file. Doing so updates the file description with the file’s new inode number if it exists. In most cases, the process must do this internally. Otherwise, we may have to restart it.

This solution, though simple, can be non-trivial with mounted shared directories.

The most common scenarios where stale file handles are not refreshed are NFS or CIFS mounted shared directories. Implementations of such protocols often lack standardized cache and file handle synchronization. Such scenarios usually require remounts (potentially with different options) or even server process restarts. Both of these actions may cause downtime.

6. Summary

In this article, we explored stale file handles. We went through the chain of references that a file is (inode to file to file handle) and how that chain can and does break.

In conclusion, we explained how to fix stale file handles in different scenarios where they occur.