Recursive Search and Replace in Text Files

1. Overview

Searching for a pattern in a file and replacing it with a new text is a typical operation when we work in the Linux command line.

Sometimes, we want to search and replace all text files under a given directory, including the text files under its subdirectories.

In this tutorial, we’ll discuss how to search and replace in text files recursively through examples.

2. The Example

Before we go into the detail of the solutions, let’s make an example directory called myDir and create some subdirectories and files under it:

$ tree myDir
myDir
├── dir1
│   ├── dir1.1
│   │   ├── dir1.1.1
│   │   │   └── text1.1.1.txt
│   │   └── text1.1.txt
│   └── text1.txt
└── parent.txt

Let’s take a look at the content of each text file:

$ head $(find myDir -name "*.txt")
==> myDir/dir1/dir1.1/dir1.1.1/text1.1.1.txt <==
text1.1.1: I like Linux.

==> myDir/dir1/dir1.1/text1.1.txt <==
text1.1: I like Linux.

==> myDir/dir1/text1.txt <==
text1: I like Linux.

==> myDir/parent.txt <==
Parent: I like Linux.

Next, we’re going to replace “Linux” with “Linux operating system” in all text files under the myDir directory.

3. Divide and Conquer

We are not really facing an algorithm problem. However, we can borrow the “Divide and Conquer” idea to solve it.

We can divide the problem into two sub-problems:

1. Replace “Linux” with “Linux operating system” in a single file
2. Find all text files under the given directory myDir

We’ll solve the two sub-problems separately, and then we’ll combine them to solve our original problem.

Next, let’s have a look at how to divide and conquer.

4. Search and Replace in a Single Text File

Firstly, let’s solve the problem: Replace “Linux” with “Linux operating system” in a single file.

In the Linux command line, we can do text substitution and save the result back to a file in many ways.

To solve this problem, we’ll pick the sed command to do the search and replace job.

Let’s see how a straightforward sed command replaces “Linux” with “Linux operating system” in the parent.txt:

$ sed -i 's/Linux/& operating system/g' parent.txt 
$ cat parent.txt
Parent: I like Linux operating system.

Next, we need to find all text files under the myDir directory and solve the problem by passing the found files to the sed command above.

5. Search and Replace Recursively

There are many ways to find all text files under the given directory and invoke the sed command on found files.

In this section, we’ll address four different methods.

5.1. Using the find Command and the -exec <command> {} + Option

The find command can find files recursively under a given directory. Moreover, it provides an option “-exec <command> {} +”  to execute a command on all found files.

Let’s assemble our sed command and a find command to solve our problem:

$ find myDir -name '*.txt' -exec sed -i 's/Linux/& operating system/g' {} +

In the command above, the “{}” is a placeholder that will be filled by all found files. Therefore, the sed command will look like:

$ sed -i '..code..' foundFile1 foundFile2 foundFile3...foundFileN

In this way, we invoke the sed command only once instead of n times.

Now, let’s check if all text files under the directory myDir have been changed:

$ head $(find myDir -name "*.txt")
==> myDir/parent.txt <==
Parent: I like Linux operating system.

==> myDir/dir1/text1.txt <==
text1: I like Linux operating system.

==> myDir/dir1/dir1.1/text1.1.txt <==
text1.1: I like Linux operating system.

==> myDir/dir1/dir1.1/dir1.1.1/text1.1.1.txt <==
text1.1.1: I like Linux operating system.

5.2. Using the find Command and the xargs Command

In the real world, we often see the find command and the xargs command work together.

The xargs command can read the output of the find command, which is a list of files found, and then, build them into arguments of another command.

Let’s see how to combine these two commands to solve our problem:

$ find myDir -name '*.txt' | xargs sed -i 's/Linux/& operating system/g'

After we execute the command above, all text files under the myDir directory will have been changed recursively.

5.3. Using the grep Command and the xargs Command

As the name says, the find command can find files. With the –Rl option, the grep command can do it as well.

Here, the -R option tells grep to search a directory recursively, while the -l option is to skip the matching information and tell grep to print only the file names of matched files.

Let’s see how to find all files containing pattern “Linux” using the grep command:

$ grep -Rl 'Linux' myDir
myDir/parent.txt
myDir/dir1/text1.txt
myDir/dir1/dir1.1/text1.1.txt
myDir/dir1/dir1.1/dir1.1.1/text1.1.1.txt

Now, to solve our problem, we just pipe this result to the xargs command:

$ grep -Rl 'Linux' myDir | xargs sed -i 's/Linux/& operating system/g'

Once again, all occurrences of “Linux” are replaced with “Linux operating system” in all text files under the myDir directory.

5.4. Using the Zsh Glob (**)

Zsh is a powerful and popular shell. Zsh glob supports the double-asterisk (**) glob to match files under the current directory and all its subdirectories.

Let’s see how to list all text files recursively under the myDir directory with Zsh:

(zsh)$ ls -1 myDir/**/*.txt
myDir/dir1/dir1.1/dir1.1.1/text1.1.1.txt
myDir/dir1/dir1.1/text1.1.txt
myDir/dir1/text1.txt
myDir/parent.txt

Therefore, we can solve our problem much simpler with Zsh:

(zsh)$ sed -i 's/Linux/& operating system/g' myDir/**/*.txt

We see that the sed command alone can solve the problem.

Similarly, we can use the same glob to check if all occurrences of “Linux” in all text files are replaced:

(zsh)$ head myDir/**/*.txt
==> myDir/parent.txt <==
Parent: I like Linux operating system.

==> myDir/dir1/text1.txt <==
text1: I like Linux operating system.

==> myDir/dir1/dir1.1/text1.1.txt <==
text1.1: I like Linux operating system.

==> myDir/dir1/dir1.1/dir1.1.1/text1.1.1.txt <==
text1.1.1: I like Linux operating system.

6. Conclusion

In this article, we addressed four different approaches to search and replace recursively under a given directory.

The basic idea is first finding all text files and then passing the file list to a text substitution command like sed.

The find, grep, and xargs commands are pretty convenient ways to solve the problem.

However, if we are using Zsh, we shouldn’t forget the handy double-asterisk (**) glob. It can solve the problem in a more straightforward way.