How to Identify Duplicate Lines in a File Without Deleting Them

1. Introduction

When using Linux and during general data science and manipulation, we might often encounter situations where we need to work with large files that may contain duplicate data. Identifying duplicates by line can be essential for understanding the activities and tracking recurring issues to make better decisions.

For example, duplicate lines may indicate repeated requests, errors, or security threats. By identifying these duplicate lines, we can refine the data and optimize system performance.

In this tutorial, we’ll explore various commands like sort, uniq, and awk, as well as a Bash script to identify duplicate lines in a file without deleting them.

2. Sample File for Demonstration

Let’s consider a sample file, duplicate_check.txt, and display its content:

$ cat duplicate_check.txt
apples
oranges
bananas
apples
Watermelon
oranges
mangoes
apples

After reviewing the content, we can observe that some lines are repeating, e.g., apples and oranges.

However, as discussed earlier, if a file contains hundreds or thousands of lines, it may be difficult to track duplicate lines manually.

So, the methods we discuss can mostly be applied to files of any size with minor caveats.

3. Using the sort Command

We can use the sort command to arrange the content of a file or stream in alphabetical order, making it easier to spot duplicate lines:

$ sort duplicate_check.txt
apples
apples
apples
bananas
mangoes
oranges
oranges
watermelon

In the output, we can see that the duplicate lines are now grouped.

However, it’s worth noting that the sort command also shows those lines which aren’t duplicates. So, we still manually examine the output and look for repeated lines to identify duplicates.

4. Using the uniq Command

The primary purpose of the uniq command is to identify and remove duplicate lines from a file. However, the uniq command only works with consecutive duplicate lines. Therefore, we should combine it with the sort command for effective use.

For example, we can use the -d option to display only the duplicate lines by piping the input from sort:

$ sort duplicate_check.txt | uniq -d
apples
oranges

Here, the pipe operator (|) redirects the output of the sort command to the uniq command.

In the output, only the duplicate lines are displayed, but we don’t see how many times each line appears.

For this purpose, we can combine the -c and -d options to display the count of duplicate lines:

$ sort duplicate_check.txt | uniq -cd
3 apples
2 oranges

Now we can see the exact count of each duplicate line. In this case, these are 3 for apples and 2 for oranges.

5. Using the awk Command

The awk command is a powerful tool that we can use to perform complex text-processing tasks such as identifying duplicate lines:

$ awk '{ count[$0]++ } END { for (line in count) if (count[line] > 1) print line }' duplicate_check.txt
apples
oranges

Now let’s understand the command:

• { count[$0]++ } counts the occurrence of each line in the file
• END { … } is executed after the entire file has been processed
• for (line in count) iterates over the associative array count, where line represents each unique line in a file
• if (count[line] > 1) check if the count of a line is greater than one, indicating it as a duplicate line
• print line prints the output if the condition is true

To get the exact count of each duplicate line, we can make minor changes to the above awk command:

$ awk '{ count[$0]++ } END { for (line in count) if (count[line] > 1) print count[line] " " line }' duplicate_check.txt
3 apples
2 oranges

In this command, we added print count[line] ” ” line which prints the number of the line, followed by space and then the line itself, if the condition is true. Effectively, we code the behavior of the uniq command we previously demonstrated.

6. Using a Bash Script

In addition to the above commands, we can also create a Bash script to identify duplicate lines in a file:

$ nano find_duplicate.sh
#!/bin/bash

read -p "Enter the file name: " file_name

declare -A lines

while IFS= read -r line; do
  (( lines[$line]++ ))
done < "$file_name"

for line in "${!lines[@]}"; do
  if [ ${lines[$line]} -gt 1 ]; then
    echo "${lines[$line]} $line"
  fi
done

Now let’s understand the script:

• read -p “Enter the file name: ” file_name prompts the user to enter the file name (later stored in the file_name variable) they want to check for duplicate lines
• declare -A lines declare an associative array called lines to store each unique line from the file and their corresponding counts
• while IFS= read -r line; do … done < “$file_name” reads the specified file line by line, increment counter in the lines array for each line, and stores the count in memory for later use
• for line in “${!lines[@]}”; do … done iterates over the lines array and checks if the count of each line is greater than one and then prints the count and line if it’s true

Moreover, we also assume that the file is in the same directory as the Bash script. Otherwise, we need to provide the complete path to the file.

7. Conclusion

In this article, we discussed several commands like sort, uniq, awk, and Bash scripting to identify duplicate lines in a file.

The sort command is simple but may not be effective for larger files. The uniq command efficiently identifies duplicates but requires a combination with the sort command.

Finally, both awk and Bash scripting provide flexible solutions but require a good understanding of their syntax.