1. Overview
curl is a useful command-line tool that we can use to transfer data over a computer network. In this tutorial, we’ll look at a few ways to display the request message header that curl sends to a destination server.
We tested the code using 64-bit curl 7.64.0 running on 64-bit Debian 10.10 (Buster) with GNU bash 5.0.3.
2. Using curl
Using curl is pretty straightforward:
$ curl https://baeldung.com
<!DOCTYPE html>
<!--[if lt IE 7]<html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7] <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8] <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--<html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
...2.1. Displaying Raw Message
We’ll pass -v or –verbose to display the raw messages that we send to or receive from the opposite end. It will also show a bunch of TCP/IP protocol messaging, including the SSL handshaking process:
$ curl -LvA "Mozilla" http://baeldung.com
* Expire in 0 ms for 6 (transfer 0x55780f420fb0)
...
* Connected to baeldung.com (104.26.12.74) port 80 (#0)
> GET / HTTP/1.1
> Host: baeldung.com
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Sun, 05 Sep 2021 15:56:24 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Sun, 05 Sep 2021 16:56:24 GMT
< Location: https://www.baeldung.com/
...
* Issue another request to this URL: 'https://baeldung.com/'
* Expire in 1 ms for 1 (transfer 0x55780f420fb0)
...
* Connected to baeldung.com (104.26.12.74) port 443 (#1)
...
> GET / HTTP/2
> Host: baeldung.com
> User-Agent: curl/7.64.0
> Accept: */*
>
...
< HTTP/2 200
< date: Sun, 05 Sep 2021 15:56:24 GMT
< content-type: text/html; charset=UTF-8
...In the above output, the destination server responds with 301 for the first request message, meaning it redirects the requested page to another location. Passing the -L or –location parameter will make curl automatically follow the redirect(s).
The -A or –user-agent parameter means user agent. Some web servers require it, some don’t.
2.2. Displaying Message Header Only
To display message header only, we’ll pass the -s or –silent parameter to suppress the progress display, and -o /dev/null to suppress the received body:
$ curl -vsA "Mozilla" -o /dev/null https://www.baeldung.com
* Expire in 0 ms for 6 (transfer 0x5589fa217fb0)
...
* Connected to baeldung.com (104.26.12.74) port 443 (#0)
...
> GET / HTTP/2
> Host: baeldung.com
> User-Agent: curl/7.64.0
> Accept: */*
>
...
< HTTP/2 200
< date: Sun, 05 Sep 2021 16:23:50 GMT
< content-type: text/html; charset=UTF-8
...In the output above, it still displays the TCP/IP and SSL stuff.
curl puts a ‘>’ character preceding the request message, and ‘<‘ preceding the response message, and since the TCP/IP and SSL log messages usually don’t have those characters, we can safely get the lines that have them.
Let’s use the grep command with pattern ‘>\|<‘, which means that we want to get the output from the curl command that contains only a ‘>’ or ‘<‘ character:
$ curl -vsA "Mozilla" -o /dev/null https://www.baeldung.com 2>&1 | grep '>\|<'
> GET / HTTP/2
> Host: www.baeldung.com
> User-Agent: Mozilla
> Accept: */*
>
< HTTP/2 200
< date: Sun, 05 Sep 2021 16:27:42 GMT
< content-type: text/html; charset=UTF-8
...The 2>&1 means that we redirect stderr to stdout, then we pipe/direct them to grep command.
We can also use the sed command to remove the TCP/IP log messages:
$ curl -vsA "Mozilla" -o /dev/null https://www.baeldung.com 2>&1 | sed '/^[* {}]/d'
> GET / HTTP/2
> Host: www.baeldung.com
> User-Agent: Mozilla
> Accept: */*
>
< HTTP/2 200
< date: Sun, 05 Sep 2021 16:30:03 GMT
< content-type: text/html; charset=UTF-8
...Description:
- /^[* {}]/d: remove line(s) starting with ‘*’, space, ‘{‘, or ‘}’ character
2.3. Displaying Request Message Header Only
To display request message header only, we can still use grep to get the line(s) that has the ‘>’ character:
$ curl -vs https://www.baeldung.com 2>&1 >/dev/null | grep '>'
> GET / HTTP/2
> Host: www.baeldung.com
> User-Agent: curl/7.64.0
> Accept: */*
>Or sed, by removing line(s) that starts with ‘*’, space, ‘{‘, ‘}’, or ‘<‘ character:
$ curl -vso /dev/null https://www.baeldung.com 2>&1 | sed '/^[* {}<]/d'
> GET / HTTP/2
> Host: www.baeldung.com
> User-Agent: curl/7.64.0
> Accept: */*
>2.4. Displaying Clean Request Message Header Only
Let’s clean up the output from curl‘s indenting and formatting styles by using grep and cut commands:
$ curl -vs https://www.baeldung.com 2>&1 >/dev/null | grep '>' | cut -c1-2 --complement
GET / HTTP/2
Host: www.baeldung.com
User-Agent: curl/7.64.0
Accept: */*
We pipe the output from curl command to grep command, where grep will filter the line(s) containing the ‘>’ character. Then, we pipe the output from the grep command to the cut command, where cut will remove the first two characters from each line.
We can also clean up the output by using sed:
$ curl -vso /dev/null https://www.baeldung.com 2>&1 | sed '/^[* {}<]/d; s/> //;'
GET / HTTP/2
Host: www.baeldung.com
User-Agent: curl/7.64.0
Accept: */*
We pipe the output from the curl command to the sed command, where sed will remove the line(s) that starts with ‘*’, space, ‘{‘, ‘}’, and ‘<‘ characters, and will also remove the ‘> ‘ pattern from each line.
3. Conclusion
In this article, we learned a few ways to display the request message header that the curl program sends to the destination server by using grep and cut commands, or simply the sed command.