1. Overview
Sometimes, we may want to filter the standard error of a command without modifying the standard output. For instance, we may want to use grep to process the standard error of another command.
When we use pipes to filter a process’ output, we can filter the standard output alone or the standard output and error. However, we can’t use pipes to filter only the standard error.
In this article, we’ll learn two approaches to overcome this limitation.
2. Using Process Substitution
With process substitution, bash sets a filename as an alias for a command’s input or output. If we write >(command), then we’ll get a filename representing the command’s input. Alternatively, we can write <(command) to refer to the command’s output.
Let’s see how it works using grep:
$ echo This parameter: >(grep baeldung) is a filename to the command\'s input.
This parameter: /dev/fd/63 is a filename to the command's input.As we can see, bash replaced the command’s input with the file /dev/fd/63.
Let’s combine this with redirections to write to that “file”:
$ echo -e "line 1: baeldung
line 2: linux" > >(grep baeldung)
line 1: baeldungIn this case, echo‘s standard output was redirected to a file that connects to grep‘s standard input.
In Linux, there are 3 standard file descriptors. The standard input (STDIN) is the file descriptor 0, the standard output (STDOUT) is the file descriptor 1, and the standard error (STDERR) is the file descriptor 2. Using this concept, we can also redirect the file descriptor 2 instead to filter only the standard error.
When we run curl -v http://www.example.com, the HTML response is printed to the standard output and the HTTP headers to the standard error. Let’s redirect the standard error through grep to filter the Accept HTTP header:
$ curl -v http://www.example.com 2> >(grep Accept:)
> Accept: */*
<!doctype html>
<html>
<head>
<title>Example Domain</title>
...As we can see, grep only filtered the standard error, leaving the HTML content intact.
We have to take into account that grep‘s output is the standard output. However, we’re feeding grep with curl‘s standard error. This can be a disadvantage, for instance, if we append more filters. Let’s add >&2 to the grep filter to separate the standard error from the standard output:
$ curl -v http://www.example.com 2> >(grep Accept: >&2)This helps to prevent mixing the outputs, and grep will print the Accept header to the standard error. With that, we can use two different filters, one for the standard output and another one for the standard error.
Let’s add grep “<title>” to filter the standard output:
$ curl -v http://www.example.com > >(grep "<title>") 2> >(grep Accept: >&2)
> Accept: */*
<title>Example Domain</title>As we can see, we managed to pass the standard output through one filter, grep “<title>”, and the standard error through a different filter, grep Accept:. Notice we redirected the output from grep Accept: to the standard error. If we didn’t do that, grep “<title>” will also filter the output from grep Accept:.
3. Swapping the File Descriptors
Another way to filter only the standard error is to swap the file descriptors. We can interchange the standard output and standard error, allowing us to use pipes to filter only the standard output, which originally was the standard error. This swap is done in 3 steps using redirections.
First, we’ll create a new file descriptor 3, as a copy of the standard output, with 3>&1. Next, we’ll make the standard output a copy of the standard error using >&2. Finally, we’ll make the standard error a copy of the file descriptor 3 with 2>&3. Remember that file descriptor 3 is a copy of the standard output. As we have created a new file descriptor 3, we’ll then close it with 3>&-.
Let’s repeat the previous curl example, but this time we swap the file descriptors:
$ curl -v http://www.example.com 3>&1 >&2 2>&3 3>&- | grep Accept:
> Accept: */*
<!doctype html>
<html>
<head>
<title>Example Domain</title>
...With that, we’ve received the HTML content intact, and we filtered only the Accept HTTP header.
Since we swapped the file files descriptors, curl prints the HTML content to the standard error and the HTTP headers to the standard output. This is the inverse of the original behavior. If we want to keep the original behavior, we have to swap the file descriptors again.
To return the file descriptors to their original meaning, we can surround the command and filter in a subshell, and then we’ll swap the file descriptors once more:
$ (curl -v http://www.example.com 3>&1 >&2 2>&3 3>&- | grep Accept:) 3>&1 >&2 2>&3 3>&-Now, we can add another pipe to filter the original standard output.
Let’s add grep to filter the <title> section:
$ (curl -v http://www.example.com 3>&1 >&2 2>&3 3>&- | grep Accept:) 3>&1 >&2 2>&3 3>&- | grep "<title>"
> Accept: */*
<title>Example Domain</title>As we can see, the first grep filtered only the HTTP headers. Then, the last grep filtered only the HTML content.
4. Conclusion
In this article, we saw two ways to filter only the standard error.
We first looked at using process substitution to redirect the standard error to another process. Then, we swapped file descriptors combined with pipes.
Additionally, we saw how to use a filter for the standard error and another different filter for the standard output.