How to Use ls on Very Large Directories

1. Overview

In Linux, ls is a commonly used command that allows users to list the contents of a directory. It works efficiently on small directories. However, listing large directories containing thousands or millions of files can present some challenges, including slow performance and overwhelming output.

In this tutorial, we’ll discuss optimizing ls to handle large directories more efficiently.

2. Disable Sorting

By default, ls sort files alphabetically before displaying them. This sorting process can be time-consuming for large directories. We can disable the sorting process using the -U option:

$ ls -U
unicode-match-property-value-ecmascript  randombyte  jsesc  es-shim-unscopables
lodash.uniq  esprima  flat-cache  language-subtag-registry
...

In the above command, the -U option instructs ls to list the contents of the current directory in their default order, without sorting.

3. Listing Only a Limited Number of Files

Viewing all files at once can be overwhelming when dealing with large directories. To manage this, we can combine ls with commands such as head and tail to list only a limited number of files.

3.1. Combining ls With the head Command

By piping the output of ls into the head command, we can display only the first few lines without listing the entire content in the directory:

$ ls | head
abab
accepts
acorn
...

Here, the head command displays the first 10 lines by default. However, we can customize the number of entries we want to display using the -n option:

$ ls | head -n 15
abab
accepts
...
agent-base
ajv

In this example, we display the first 15 files or directories in the current directory.

3.2. Combining ls With the tail Command

While the head command allows us to list the first few lines, the tail command displays the last few lines or entries. It helps us view files or directories at the end of a long list.

Similar to the head command, the tail command also displays the last 10 entries by default, but we can customize it to display a specific number of files or directories:

$ ls | tail -n 25
workbox-navigation-preload
workbox-precaching
workbox-range-requests
...

4. Using ls With Pagination

When we need to view a large directory’s entire contents but don’t want to overwhelm the screen, we can use pagination. We can pipe the output of ls to the less or more commands to achieve this. By using these commands, we can view the output one screen at a time.

4.1. Using less

The less command allows us to view the ls output in a scrollable fashion. It loads the output one screen at a time and we can navigate the list using the keyboard keys:

$ ls | less

To scroll forward, we press the Spacebar or PgDn key to scroll down one page at a time. Next, to scroll backward, we press the b or PgUp key to scroll up.

Furthermore, we can search for a specific file or directory in the list, by typing the forward-slash symbol / followed by the search item and then pressing Enter. To move to the next occurrence of the search term, we press the n key.

Lastly, to return to the shell, we press the q key.

4.2. Using more

The more command also allows us to view one screen at a time, but it’s more limited in functionality compared to less, as it only allows forward navigation:

$ ls | more

Here, we press the Spacebar key to scroll down one page at a time. Additionally, we can press the Enter key to scroll one line at a time. To quit and return to the shell, we press the q key.

5. Limit the Output Using Wildcards

When working with large directories, sometimes we don’t need to list all the contents in the directory. Using wildcards, we can filter the output of ls by matching filenames based on a specific pattern. This limits the amount of data we need to look through.

5.1. List Files With a Specific Extension

In large directories, files often have various extensions. In this case, we can limit the output to only files of a particular type. By using shell wildcards, we can instruct ls to list only files that match a specific pattern.

For instance, let’s match all files that contain the .js extension:

$ ls *.js
acorn.js  browser.js  doUntil.js  findSeries.js  helpers.js

Above, we use the * wildcard to list all files in the current directory that end with the .js extension.

5.2. List Files or Directories With a Specific Pattern in Their Name

We can also find files or directories whose names match a specific pattern. To demonstrate, let’s list all files or directories that contain find in their names:

$ ls find*
find.js  findLimit.js  findSeries.js

find-cache-dir:
index.js  license  package.json  readme.md
...

In this example, we list all files and directories in the current directory whose names start with the prefix find. To clarify, when we match a directory whose name starts with find, ls lists the contents of the directory as well.

5.3. List Only Directories

In large directories containing a mix of files and directories, we may only want to list the directories. We can achieve this by using the -d option along with a wildcard. For example, let’s list all directories in the current directory:

$ ls -d */
abab/  graceful-fs/  postcss-merge-longhand/  accepts/ 
...

In the above command, the -d option instructs ls to list directory names instead of their contents, while */ represents a wildcard that matches only directories.

6. Conclusion

In this article, we discussed optimizing the ls command to work efficiently in large directories. First, we explored how to disable sorting using the -U option to improve performance. Second, we used the head and tail commands to limit the number of files displayed. Next, we used the less and more commands to paginate the output. Lastly, we used wildcards to filter files and directories that matched a specific pattern.