1. Introduction

In this tutorial, we’re going to take a look at how we can harness the power of built-in Linux commands to search for and replace a string of characters quickly and easily. This is a very handy technique for when we need to update all occurrences of a particular string in a large number of files.

For example, one typical scenario could be when we want to update the copyright notice in a collection of static HTML files.

2. Search and Replace With sed

The first command that we’re going to look at is sed, a powerful stream editor that is useful for performing basic transformations on an input stream. Take a look at our Guide to Stream Redirections in Linux for a refresher on what streams are.

Using sed, we’re able to quickly and easily find and replace a set of characters within a file.

Let’s begin by creating a test file to use in our examples. We’ll use a here document to create this test file quickly:

$ cat <<-EOF > test.txt
This is a sample file created in 2019 to demonstrate character substitution. 
It will be used in 2019. 
Linux has a wide array of tools for us to use to achieve this.
2019 is a leap year.
EOF

Let’s confirm that we created the test file successfully:

$ cat test.txt
This is a sample file created in 2019 to demonstrate character substitution. 
It will be used in 2019. 
Linux has a wide array of tools for us to use to achieve this.
2019 is a leap year.

Now everything is as it should be, so let’s begin using sed.

2.1. Find and Replace the First Occurrence

We’re going to use sed to manipulate our test file and update the year to the current year.

An easy way to remember the syntax of the sed command for replacing the first occurrence of a word or term is:

$ sed -i 's/{OLD_TERM}/{NEW_TERM}/' {file}

Now let’s apply this command to our example:

$ sed -i 's/2019/2020/' test.txt

And now, we print the contents of our file to confirm that we only replaced the first occurrence of “2019”:

$ cat test.txt
This is a sample file created in 2020 to demonstrate character substitution. 
It will be used in 2019. 
Linux has a wide array of tools for us to use to achieve this.
2019 is a leap year.

Let’s break down our statement to get a deeper understanding of how this works:

  • First, we pass the -i option to instruct sed to make the changes inside our test.txt file. By default, sed prints changes to the terminal
  • Next is the regular expression that specifies what we want to change
    • We start the expression with the letter “s” so that sed knows we’re performing a substitution
    • Next up, we have the string “2019” which is the value for the OLD_TERM placeholder. This is the section of text that we want to replace
    • Following that is the value of the NEW_TERM placeholder and that’s “2020”
    • We separated these parameters using forward slashes (/)

As we can see, sed is a quick and easy way to perform search and replace operations.

2.2. Find and Replace All Occurrences

By default, sed only replaces the first occurrence that it finds. We can easily override this behavior to replace all instances within a file.

Let’s see the expression that we can use to achieve this:

$ sed -i 's/{OLD_TERM}/{NEW_TERM}/g' {file}

We’ve added “g” to the end of our search expression. This instructs sed to replace all occurrences globally.

So let’s apply this to our text file:

$ sed -i 's/2019/2020/g' test.txt

Now let’s print out the contents of the test.txt again. This time we’ve replaced all occurrences of “2019” with “2020”:

$ cat test.txt
This is a sample file created in 2020 to demonstrate character substitution.
It will be used in 2020.
Linux has a wide array of tools for us to use to achieve this.
2020 is a leap year.

3. Search and Replace With awk

In this section, we’ll take a look at awk. A powerful scripting language that is designed for text processing and is often used for data extraction and reporting purposes. As is common with other Linux utilities, awk can perform operations on both streams and files.

Let’s look at how awk performs find-and-replace operations on our test.txt sample file:

$ awk '{sub(/{OLD_TERM}/,{NEW_TERM}); print}' {file}

In this statement, we’re invoking awk and sending in a list of tasks for awk to perform.

Let’s break this down:

  • The first task in our list is a substitute, which we write as sub
  • Sub directs awk to find all occurrences of OLD_TERM and replace them with NEW_TERM
  • Our next instruction directs awk to print the output to the standard output stream; the console
  • Finally, we have the name of the file that awk will be working on

A notable difference with sed is that awk will not perform an in-place substitution; meaning that the updates will not be made inside the file.

We’ll tackle that shortly but for now, let’s see this in action:

$ awk '{sub(/2019/,2020); print}' test.txt

Our resulting output confirms that the replacement has been carried out correctly:

This is a sample file created in 2020 to demonstrate character substitution.
It will be used in 2020.
Linux has a wide array of tools for us to use to achieve this.

As expected, awk has replaced all instances of “2019” with “2020”.

However, our output is directed to the console instead of updating our original file. We have a trick for taking care of that.

We’ll use our knowledge of streams and stream redirection to update our awk find-and-replace command and use it to update our original file in a single step:

$ awk '{sub(/2019/,2020); print . "text.txt" }' test.txt

This time nothing is printed to the console but let’s dump the contents of test.txt to see what’s happened:

$ cat test.txt
This is a sample file created in 2020 to demonstrate character substitution.
It will be used in 2020.
Linux has a wide array of tools for us to use to achieve this.

Here we’re seeing that the output of the awk command was correctly redirected to our input file thereby updating it for us.

4. Conclusion

In this tutorial, we looked at how we could use two of the most common Linux utilities to find and replace a string of characters in a single file or a set of files without manually having to edit each individual file.

These utilities are extremely powerful and come in handy for a variety of day-to-day tasks whilst working on the Linux command line. Be sure to have a look at the documentation of these commands to learn more about them.

Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments