Monitoring HTTP Requests on a Network Interface in Real Time

1. Overview

System administrators and end-users can monitor HTTP requests on a network interface for many purposes (like debugging).

In this tutorial, we’ll discuss two programs we can use to monitor HTTP requests and responses on a network interface in real-time.

2. Using tcpflow

The first program that we can use is tcpflow.

2.1. Installing tcpflow

Many Linux distros have tcpflow in their repositories. So, we can install tcpflow using package managers:

# On Debian/Ubuntu/etc
$ sudo apt install tcpflow

# On Fedora/RedHat/CentOS/etc
$ sudo dnf install tcpflow

After that, let’s check if it’s actually installed:

$ tcpflow --version
TCPFLOW 1.5.1

We’ve successfully installed tcpflow.

2.2. Listing All Network Interfaces

Before monitoring, we should find the network interface that we want to monitor. So, let’s call ifconfig to list all network interfaces:

$ ifconfig -a
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
...
wlp0s20f3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.24  netmask 255.255.255.0  broadcast
...

Let’s monitor wlp0s20f3.

2.3. Monitoring a Network Interface

Now that we’ve found the interface, we can monitor it using tcpflow:

$ sudo tcpflow -p -c -i wlp0s20f3 port 80 | grep -oE '(GET|POST) .* HTTP/1.[01]|Host: .*'
reportfilename: ./report.xml
tcpflow: listening on wlp0s20f3
GET /alexlarsson/flatpak/ubuntu/dists/focal/InRelease HTTP/1.1

GET /mirrors.txt HTTP/1.1

In the above command:

• -p disables promiscuous mode
• -c means only print the output to the console and don’t create files
• -i specifies the network interface
•  grep receives the output of tcpflow
• -o means show only the matching parts of the lines that match the pattern
• -E means the pattern is an extended regular expression (ERE)

We can see that tcpflow started listening on wlp0s20f3 and found two GET requests.

Moreover, we can add more HTTP methods (like HEAD, PUT, etc) to the grep pattern.

3. Using httpry

In addition, we can use httpry to monitor a network interface.

3.1. Installing httpry

To install httpry, we must build it from the source since it is not available in repositories:

$ git clone https://github.com/jbittel/httpry.git
$ cd httpry
$ make
$ sudo make install

After that, let’s check if it’s installed:

$ httpry -h
httpry version 0.1.8 -- HTTP logging and information retrieval tool
Copyright (c) 2005-2014 Jason Bittel <[email protected]>

We’ve successfully installed httpry from the source.

3.2. Monitoring a Network Interface

To monitor a network interface using httpry, we can run:

$ sudo httpry -i wlp0s20f3
httpry version 0.1.8 -- HTTP logging and information retrieval tool
Copyright (c) 2005-2014 Jason Bittel <[email protected]>
Starting capture on wlp0s20f3 interface
2022-06-22 16:38:12.166	192.168.1.24	172.217.17.238	>	GET	google.com	/	HTTP/1.1	-	-
2022-06-22 16:38:12.199	172.217.17.238	192.168.1.24	<	-	-	-	HTTP/1.0	400	Bad Request
2022-06-22 16:38:23.090	192.168.1.24	172.217.17.238	>	POST	google.com	/	HTTP/1.1	-	-
2022-06-22 16:38:23.163	172.217.17.238	192.168.1.24	<	-	-	-	HTTP/1.1	405	Method Not Allowed

Here, wlp0s20f3 is the network interface that we wanted to monitor. Further, we can see that httpry detected one GET request, one POST request, and two server responses.

3.3. Monitoring Particular HTTP Methods

We can also specify the HTTP request methods that we want to monitor:

$ sudo httpry -i wlp0s20f3 -m post
httpry version 0.1.8 -- HTTP logging and information retrieval tool
Copyright (c) 2005-2014 Jason Bittel <[email protected]>
Starting capture on wlp0s20f3 interface
2022-06-22 16:26:47.516	192.168.1.24	172.217.17.238	>	POST	google.com	/	HTTP/1.1	-	-
2022-06-22 16:26:47.582	172.217.17.238	192.168.1.24	<	-	-	-	HTTP/1.1	405	Method Not Allowed

This time, httpry detected one POST request and one server response.

3.4. Saving the Output

To save the captured packets in a human-readable format, we can add -o:

$ sudo httpry -i wlp0s20f3 -o human.txt

After that, human.txt will contain the captured packets.

Moreover, we can save the captured packets in a binary file that can be read by httpry later using the -b option:

$ sudo httpry -i wlp0s20f3 -b binary.o

After that, we can use httpry itself to read the output:

$ httpry -r binary.o
...
2022-06-19 05:23:21.771	192.168.1.24	91.189.90.8	>	GET	mirrors.ubuntu.com	/mirrors.txt	HTTP/1.1	-	-
2022-06-19 05:23:21.822	192.168.1.24	185.125.190.52	>	GET	ppa.launchpad.net	/alexlarsson/flatpak/ubuntu/dists/focal/InRelease	HTTP/1.1	-	-
2022-06-19 05:23:21.832	91.189.90.8	192.168.1.24	<	-	-	-	HTTP/1.1	416	Requested Range Not Satisfiable

httpry read the binary file and printed the captured packets on the screen.

4. Conclusion

Users may need to monitor a specific network interface for many reasons. Here, we learned how to use two programs that we can use to monitor network interfaces in real-time.