Pipes and Redirection in Linux

1. Introduction

Most shells offer the ability to alter the way that application input and output flows. This can direct output away from the terminal and into files or other applications or otherwise read input from files instead of the terminal.

In this tutorial we’ll discuss the concepts of pipes and redirection in Linux.

2. Standard Input and Output

Before we understand how redirection works in shells, we first need to understand the standard input and output process.

All applications have three unique file descriptors that connect them to the outside world. These are referred to as Standard Input, or stdin; Standard Output, or stdout; and Standard Error, or stderr.

Standard input is the default mechanism for getting input into an interactive program. This is typically a direct link to the keyboard when running directly in a terminal and isn’t connected to anything otherwise.

Standard output is the default mechanism for writing output from a program. This is typically a link to the output terminal but is often buffered for performance reasons. This can be especially important when the program is running over a slow network link.

The standard error is an alternative mechanism for writing output from a program. This is typically a different link to the same output terminal but is unbuffered so that we can write errors immediately while the normal output can be written one line at a time.

3. Redirecting Into Files

One common need when we run applications is to direct the output into a file instead of the terminal. Specifically, what we’re doing here is replacing the standard output file descriptor with one that goes where we want: In this case, a file.

This is typically done with the > operator between the application to run and the file to write the output into. For example, we can send the output of the ls command into a file called files as follows:

$ ls > files

We can do this after any command, including any needed arguments:

$ ls -1 *.txt > text-files

3.1. Appending To Files

When we use >, then we are writing the output to the file, replacing all of the contents. If needed, we can also append to the file using >> instead:

$ ls -1 *.txt > files
$ ls -1 *.text >> files
$ ls -1 *.log >> files

We can use this to build up files using all manner of output if we need to, including just using echo to exact output lines:

$ echo Text Files > files
$ ls -1 *.txt >> files
$ echo Log Files >> files
$ ls -1 *.log >> files

3.2. Redirecting Standard Error

On occasion, we need to redirect standard error instead of standard output. This works in the same way, but we need to specify the exact file descriptor.

All three of the standard file descriptors have ID values, as defined in the POSIX specification and used in the C language. Standard input is 0, standard output is 1, and the standard error is 2.

When we use the redirect operators, by default, this applies to standard output. We can explicitly specify the file descriptor to redirect, though, by prefixing it with the file descriptor ID.

For example, to redirect standard error from the cat command, we would use 2>:

$ cat does-not-exist 2> log

We can be consistent and use 1> to redirect standard output if we wish, though this is identical to what we saw earlier.

We can also use &> to redirect both standard output and standard error at the same time:

$ ls -1 &> log

This sends all output from the command, regardless of which file descriptor it is on, into the same file. This only works in certain shells, though – for example, we can use this in bash or zsh, but not in sh.

3.3. Redirecting into File Descriptors

Sometimes we want to redirect into a file descriptor instead of a file. We can achieve this by using the file descriptor ID, prefixed by &, in place of the filename. For example, we can redirect into standard error by using >&2:

$ echo Standard Output >&1
$ echo Standard Error >&2

We can combine this with the above to combine file descriptors by redirecting from one file descriptor into another. A common construct is to combine standard error into standard output so that both can be used together. We achieve this using 2>&1 – literally redirecting file descriptor 2 into a file descriptor 1:

# ls -1 2>&1

We can sometimes use this to create new file descriptors simply by using new IDs. This file descriptor must already have been used elsewhere first; otherwise, it is an error. Most often, this is used as a file descriptor source first.

For example, we can swap standard output and standard error by going via a third file descriptor, using 3>&2 2>&1 1>&3:

$ ls -1 3>&2 2>&1 1>&3

This construct doesn’t work correctly in all shells. In bash, the end result is that standard output and standard error are directly swapped. In zsh the result is that both standard output and standard error have ended up on standard output instead.

3.4. Redirecting Multiple File Descriptors

We can easily combine the above to redirect standard output and standard error at the same time. This mostly works exactly as we would expect – we simply combine the two different redirects on the same command:

$ ls -1 > stdout.log 2> stderr.log

This won’t work as desired if we try to redirect both file descriptors into the same file. What happens here is that both file descriptors are redirected individually, and whichever comes second wins, rather than combining both into the same file.

If we want to redirect both into the same file, then we can use &> as we saw above, or else we can use file descriptor combination operators. If we wish to use the file descriptor combination operators, then we must do this after we have redirected into the file, or else only the standard output gets redirected:

$ ls -1 > log 2>&1

4. Reading From Files

Sometimes we also want to achieve the opposite – redirecting a file into an application. We do this using the < operator, and the contents of the file will replace the standard input for the application:

$ wc < /usr/share/dict/words
  235886  235886 2493109

When we do this, the only input that the application can receive comes from this source, and it will all happen immediately. It’s effectively the same as when the user types out the entire contents of the file at the very start of the application.

However, the end of the file is signaled to the application as well, which many applications can use to stop processing.

5. Piping Between Applications

The final action that we can perform is to direct the output of one application into another one. This is commonly referred to as piping and uses the | operator instead:

$ ls | wc
      11      11     138

This directly connects the standard output of our first application to the standard input of the second one and then lets the data flow directly between them.

5.1. Handling Standard Error

The standard error isn’t connected by default, so we’ll still get anything written to that file descriptor appearing in the console. This is by design since the standard error is designed for error reporting and not for normal application output.

If we want to redirect standard error as well, then we can use the technique from above to first redirect it into standard output and then pipe into the next application:

$ ls i-dont-exist | wc
ls: i-dont-exist: No such file or directory
       0       0       0

$ ls i-dont-exist 2>&1 | wc
       1       7      44

If we want only to pipe standard error, then we need to swap the standard output and standard error file descriptors around, as we saw earlier.

$ ls 3>&2 2>&1 1>&3 | wc -l
some-file
       0

$ ls i-dont-exist 3>&2 2>&1 1>&3 | wc -l
       1

5.2. Combining Pipes

When piping between applications, we can also build arbitrary chains where we are piping between many applications to achieve a result:

$ docker images | cut -d' ' -f1 | tail -n +2 | sort -u | wc -l
      17

This command looks scary, but we can break this down into individual parts:

docker images – Get a list of all Docker images
cut -d’ ‘ -f1 – Cut this output to only return the first column, where columns are space-separated
tail -n +2 – Limit this to start from line 2
sort -u – Sort this list, only returning unique entries
wc -l – Count the number of lines

So we have a command here to get the number of unique docker images, ignoring the version of the image.

Many console applications are designed for exactly this use, which is why they can often consume input from standard input and write to standard output.

Certain applications also have special modes that allow for this – git, for example, has what is termed porcelain and plumbing commands, where the plumbing commands are specially designed to be combined in this manner while the porcelain commands are designed for human consumption.

5.3. Named Pipes

Another significant concept when dealing with IO in Linux is the concept of named pipes. Named pipes are a form of inter-process communication that allows data to flow between two processes, but unlike unnamed pipes, they persist in the filesystem beyond the life of the process and can be accessed as a file. This functionality can be incredibly useful when setting up consistent communication channels between different parts of your system.

6. Summary

Here, we’ve seen several techniques for redirecting the input and output of applications either between files or other applications. These are compelling techniques that can give rise to complicated results from simple inputs.

Why not see what else we can do with these?

Full Archive

About Baeldung

Administration

Filesystems

Processes

Files

Scripting

Installation

Networking

Security