How to Remove All Special Characters in Linux Text

1. Overview

In Linux, removing special characters in text is not uncommon during text analysis. For instance, we can encounter a scenario in which we may need to clean text for applications such as databases.

In this tutorial, we’ll walk through several commands for removing all special characters from Linux text. In particular, we’ll focus on tr, sed, awk, perl, and grep.

2. Understanding Special Characters

In the context of this article, special characters comprise all characters other than alphanumeric characters, for instance, whitespaces, symbols, and control characters to mention a few.

To demonstrate, let’s use the echo command to create a file with text:

$ echo -e "This is a sample text with special characters @#%^&*()_+={}[]|;:'\"<>,.?/~\` and the control characters \x01\x02\x03\x0D\x1A" > sample_text.txt

Above, we create the sample_text.txt file containing text that includes some special characters by adding the -e switch for interpreting character codes in escape sequences like \x01:

$ cat sample_text.txt
�his is a sample text with special characters @#%^&*()_+={}[]|;:'"<>,.?/~` and the control characters

This text output contains control characters. However, these characters are non-printable and meant for text formatting, device control, or special functions like line breaks, making them invisible in normal text output. In addition, control characters can appear in text due to encoding issues or file transfers between different systems.

Now, let’s make the control characters visible:

$ cat -v sample_text.txt
This is a sample text with special characters @#%^&*()_+={}[]|;:'"<>,.?/~` and the control characters ^A^B^C^M^Z

Here, we use the -v option to display the control characters as symbols in the cat output.

3. Using tr

We can use the tr command to specify a range that helps us delete all characters except alphanumerics:

$ tr -cd '[:alnum:]' < sample_text.txt > cleaned_sample_text.txt && cat -v cleaned_sample_text.txt && echo
Thisisasampletextwithspecialcharactersandthecontrolcharacters

Let’s analyze the command:

• -c  instructs tr to target characters not specified, in this case, anything other than alphanumerics
• -d deletes the targeted characters
• ‘[:alnum:]’ is the character class that represents alphanumeric characters
• < sample_text.txt reads from sample_text.txt
• > cleaned_sample_text.txt redirects the cleaned text output to the file cleaned_sample_text.txt
• && cat -v cleaned_sample_text.txt displays the cleaned output redirected to cleaned_sample_text.txt
• && echo – the echo command adds a newline to the cleaned output ensuring the output appears on a newline

This command deletes all special characters in the text including whitespaces.

4. Using sed

Alternatively, we can use the sed command with regular expressions to remove all special characters.

$ sed 's/[^a-zA-Z0-9]//g' sample_text.txt > cleaned_sample_text.txt && cat -v cleaned_sample_text.txt
Thisisasampletextwithspecialcharactersandthecontrolcharacters

Let’s analyze the command:

• ‘s/ … / … /g’  is a substitution while /g is a global flag that applies the substitution to all matches in the line
• [^a-zA-Z0-9] is a regular expression matching any character that isn’t a letter or a number
• // represents the second and third slashes with an empty string between them to ensure the command substitutes the matched characters with an empty string

Therefore, sed is another versatile tool we can use to manipulate text.

5. Using awk

We can also utilize the awk command to remove all special characters from a file or stream:

$ awk '{gsub(/[^a-zA-Z0-9]/, ""); print}' sample_text.txt > cleaned_sample_text.txt && cat -v cleaned_sample_text.txt
Thisisasampletextwithspecialcharactersandthecontrolcharacters

Let’s see the breakdown of the awk command:

• gsub(/[^a-zA-Z0-9]/, “”) is an awk function that replaces all characters not within the a-z, A-Z, and 0-9 ranges with an empty string
• print prints each modified line

This command processes the text in sample_text.txt and then displays the desired output. To clarify, awk processes text line by line and can handle complex patterns efficiently.

6. Using perl

The perl interpreter is another powerful tool we can use to remove all non-alphanumeric characters:

$ perl -pe 's/[^a-zA-Z0-9]//g' sample_text.txt > cleaned_sample_text.txt && cat -v cleaned_sample_text.txt && echo
Thisisasampletextwithspecialcharactersandthecontrolcharacters

So, perl invokes the perl interpreter:

• -p instructs perl to print each line after processing
• -e enables perl to execute a script or a code snippet from the terminal
• ‘s/[^a-zA-Z0-9]//g’ represents the code snippet to execute, in this case, the substitution expression inside the single quotes

Finally, echo displays the output on a new line.

7. Using grep

Typically, the grep command searches for patterns in text:

$ grep -o '[a-zA-Z0-9]*' sample_text.txt
This
is
a
sample
text
with
special
characters

Above, the -o option instructs grep to output only the alphanumeric sequences with each sequence in a new line. Now, we can combine grep with other commands to filter out special characters.

First, let’s use grep with sed and tr:

$ grep -o '[a-zA-Z0-9]*' sample_text.txt | sed 's/^$/ /' | tr -d '\n' > cleaned_sample_text.txt && cat -v cleaned_sample_text.txt && echo
Thisisasampletextwithspecialcharactersandthecontrolcharacters

To clarify, we can check what each command does:

• sed ‘s/^$/ /’  replaces any empty lines with a space to ensure there are no gaps in the output
• tr -d ‘\n’ removes any newlines so that everything is displayed in a single line

Here, grep employs the same idea of regular expressions.

8. Conclusion

In this article, we explored multiple approaches to remove special characters in Linux text. In particular, we looked at how to use tr, sed, awk, perl, and grep.

Each tool offers a unique approach as well as syntax for handling text manipulation. Therefore, we can select which tool to utilize depending on the task complexity, size of the dataset, or even familiarity with the syntax, offering us flexibility.

These utilities are essential for text processing whether we’re cleaning up data files, preparing text for importing into a specific application like a database, or simply formatting output.