How to Pipe Commands After xargs in a Pipeline

1. Overview

The idea of chaining commands together is at the core of the Unix philosophy of building small programs that can be combined in a pipeline to achieve complex tasks. So, the standard xargs command is a versatile tool that can pass the output of one command as arguments to a subsequent command in a pipeline. This way, xargs enables combining commands together.

In this tutorial, we’ll explore how to pipe commands after xargs to perform operations on the output of preceding commands in a pipeline.

2. Sample Task

Let’s suppose our current working directory contains three subdirectories named appA, appB, and appC. Furthermore, each one of these contains a directory named log which includes several log files.

We can view the overall directory structure using the tree command:

$ tree .
.
├── appA
│   └── log
│       ├── logA1
│       ├── logA2
│       ├── logA3
│       └── logA4
├── appB
│   └── log
│       ├── logB1
│       ├── logB2
│       ├── logB3
│       └── logB4
└── appC
    └── log
        ├── logC1
        ├── logC2
        ├── logC3
        └── logC4

6 directories, 12 files

Our objective is to find the latest two log files based on modification time in each of the log directories.

Let’s explore how we can accomplish this task with the help of xargs.

3. Standard Usage of xargs

To begin with, we can use the find command with -type d to recursively list all directories under the current directory which contain the string log in their name:

$ find . -type d -name '*log*'
./appC/log
./appA/log
./appB/log

The result is a list of *log* directory paths containing the log files we’re trying to find.

Ideally, we should be able to use ls to list the contents of each of the three directories found. Then, we can pipe the result to the tail command to select only the two most recent files within each directory:

$ find . -type d -name '*log*' | xargs ls -ltr | tail -2
-rw-r--r-- 1 sysadmin sysadmin 4 Nov 23 15:02 logC3
-rw-r--r-- 1 sysadmin sysadmin 4 Nov 23 15:02 logC4

In this case, xargs passes the output of find as arguments to the ls command. Thus, we list the contents of each of the three directory paths.

Further, we set the -l flag with ls to display the output in a long listing format. In addition, the -t flag is for sorting by modification time, while the -r flag reverses the order of the listing so that the most recent files appear at the end. Finally, we pipe the result to tail -2 to display only the last two files.

However, the result isn’t as expected since we only obtain the last two files of the ./appC/log directory. This occurs because subsequent commands operate on the result of the entire expression appearing before them in the pipeline. Therefore, the tail command selects the last two files from the entire result. Instead, we need to apply the tail command over the contents of each directory path.

Let’s see how we can modify the pipeline to achieve the required result.

4. Using the -I Flag With xargs

Generally, when xargs is followed by a command, it supplies the input it receives from stdin as arguments at the end of the command. However, we can use the -I flag with xargs to assign the input xargs receives to a placeholder string, such as {}.

This way, subsequent commands can process the input represented by the placeholder accordingly. In particular, this enables us to place the input arguments at the required positions after the echo and ls commands:

$ find . -type d -name '*log*' | xargs -I {} sh -c "echo {}; ls -ltr {} | tail -2"
./appC/log
-rw-r--r-- 1 sysadmin sysadmin 4 Nov 23 15:02 logC3
-rw-r--r-- 1 sysadmin sysadmin 4 Nov 23 15:02 logC4
./appA/log
-rw-r--r-- 1 sysadmin sysadmin 4 Nov 23 15:01 logA3
-rw-r--r-- 1 sysadmin sysadmin 4 Nov 23 15:01 logA4
./appB/log
-rw-r--r-- 1 sysadmin sysadmin 4 Nov 23 15:01 logB3
-rw-r--r-- 1 sysadmin sysadmin 4 Nov 23 15:01 logB4

The -I flag enables xargs to replace each occurrence of {} with a directory path returned by find. Therefore, the echo and ls commands appearing in quotes as part of the sh -c command execute for each directory path returned by find.

In particular, we print the directory path using echo. Then, we list its contents using ls and select the last two files using the tail command. All processing is now a separate script passed to sh.

As expected, the result shows the two most recent log files within each directory path.

5. Conclusion

In this article, we explored how to pipe commands after xargs in a pipeline.

In particular, xargs passes the output of one command as arguments to the following command in the pipeline. We can use the -I flag with xargs to position arguments via a placeholder at required locations for subsequent commands to process.