How to Identify Processes Generating UDP Traffic

Last updated: March 18, 2024

Reviewed by: Hiks Gerganov

1. Overview

Sometimes administrators or security specialists that monitor a network may find that servers are sending unexpected UDP packets. Further, to identify these packets, we need to find the processes that are generating them. However, this isn’t as easy as it might seem, because UDP is stateless. So, to use typical commands that we could apply to TCP connections such as lsof or netstat, we need to enter the command at the exact time the UDP packet is being sent, which is almost impossible to do.

In this tutorial, we’re going to explore two ways in which we can identify the processes that are generating the UDP traffic.

2. Linux Audit Service

The first way to find the PID of the processes sending the UDP traffic is by using the Linux Audit Service.

The Linux Audit Service is a component that allows us to monitor the system. To do this, we add capture rules with criteria, so the service can monitor and record the data. After that, we can search through the recorded data to find out what we want.

Indeed, there are several components to the Linux Audit Service:

  • auditd is the daemon
  • auditctl controls the daemon and rules
  • ausearch enables searching through the recorded data

So, let’s install and use auditd.

2.1. Installation

First, we check for the service by running auditctl:

$ auditctl -l
No rules

In this case, the -l flag shows the current ruleset. So far, there are no rules.

If auditd isn’t installed, we’d get a command not found error when running auditctl.

So, to install auditd on RHEL or CentOS, we’ll use the yum package manager:

$ sudo yum install audit

On the other hand, to install auditd on Ubuntu, we’ll use the apt package manager:

$ sudo apt install auditd

Next, let’s see how we can use the service to monitor UDP traffic.

2.2. Add a Rule

Before using auditd to monitor UDP traffic, we need to add a rule:

$ auditctl -a exit,always -F arch=b64 -F a0=2 -F a1\&=2 -S socket -k SOCKET

Let’s explore this command to understand the meaning of each part:

  • -a adds a new rule
  • -F adds the conditions
  • -S identifies the function or system call we want to audit
  • arch=b64 means we are using a 64-bit architecture
  • a0 and a1 are the first and second parameters of the socket system call, both include 2 in the case of UDP
  • -k makes it easier to search through the logs by adding a key

The bitwise operator near a1 prevents the loss of other values that change the behavior of the system call.

Next, we’ll try sending a UDP request and looking for it in the logs.

2.3. Search Through the Records

Now, let’s call the ping command that sends a DNS lookup request over UDP:

$ ping www.google.com

Next, let’s use the ausearch tool to view the data recorded for our rule:

$ ausearch -i -ts recent -k SOCKET

Further, we can use the grep command to make it easier to spot our ping command:

$ ausearch -i -ts recent -k SOCKET | grep ping

The -i option interprets or transforms some of the fields. For example, -i converts the UID into the corresponding user. The -ts option is the start time for the data we want to see. Here, the recent keyword means 10 minutes.

According to the system, the output might vary a bit, but it should contain some common data:

type=SYSCALL msg=audit(04/23/2023 17:59:00.104:130174) : 
arch=x86_64 syscall=socket success=yes exit=3 a0=inet a1=SOCK_RAW a2=icmp a3=0x7ffdf249264c items=0
ppid=29280 pid=32725 auid=root uid=root gid=root euid=root 
suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=6115 
comm=ping exe=/usr/bin/ping 
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=SOCKET2

Here, we inserted newlines for legibility. We can see that auditd captured the UDP requests. Let’s have a look at some of the data that can help identify their source:

  • pid is the process ID that was responsible for sending the request
  • PPID is the parent process ID, which can be useful in case the command was called from inside a script
  • uid is the ID of the user that sent the request
  • comm is the command that sent the request
  • exe is the command executable location

To delete the rule we previously added, we use the same command but replace -a with -d:

$ auditctl -d exit,always -F arch=b64 -F a0=2 -F a1\&=2 -S socket -k SOCKET

In the next part, we’ll explore another tool to monitor UDP traffic.

3. Systemtap

Another tool that we can use to monitor UDP traffic is Systemtap or stap.

Systemtap is a tool available in Linux that makes it possible to monitor and modify kernel activities without having to change the kernel code itself. It consists of a scripting language as well as a CLI. The scripts written in the Systemtap scripting language are inserted into the kernel after being compiled into C-code kernel modules.

3.1. Installation

First, we’ll make sure we’ve installed Systemtap on the machine:

$ stap -V

If the stap -V command produces an error, then we need to first install Systemtap.

To install Systemtap for Ubuntu, we’ll again use apt:

$ sudo apt-get install -y systemtap gcc

Similarly, to install Systemtap for CentOS/RHEL, we’ll use yum:

$ sudo yum install systemtap systemtap-runtime
$ sudo yum install kernel-debuginfo kernel-debuginfo-common

Importantly, to use certain tapsets, we need to install the kernel debug symbols. How we do this depends on the exact distribution.

To install the kernel debug symbols for Ubuntu, we need to get the correct sources and install the relevant packages:

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C8CAB6595FDFF622
$ codename=$(lsb_release -c | awk  '{print $2}')
$ sudo tee /etc/apt/sources.list.d/ddebs.list << EOF
deb http://ddebs.ubuntu.com/ ${codename}      main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-security main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-updates  main restricted universe multiverse
deb http://ddebs.ubuntu.com/ ${codename}-proposed main restricted universe multiverse
EOF
$ sudo apt-get update
$ sudo apt-get install linux-image-$(uname -r)-dbgsym

To install the kernel debug symbols for CentOS/RHEL, we should be able to use a single command:

$ sudo yum --enablerepo=base-debuginfo install -y kernel-debuginfo-$(uname -r)

Next, let’s see how to use Systemtap to monitor the UDP traffic.

3.2. Running a Systemtap Script

Let’s create a udp_monitor.stp file as the script to monitor UDP traffic using Systemtap:

$ cat udp_monitor.stp
probe udp.sendmsg {
    printf ("(%s) with PID: %5d sent UDP to %15s port %d\n", execname(), pid(), daddr, dport)
}

We use the probe keyword to tell Systemtap which tapset to use. So, we’ll be using the udp.sendmsg tapset to monitor UDP traffic.

Then, when a process sends a UDP request, we print several fields:

  • execname() – name of the source executable
  • pid() – process ID of the source executable
  • daddr – destination IP address
  • dport – destination port

Next, we run the script to start monitoring UDP traffic:

$ stap -v udp_monitor.stp
Pass 1: parsed user script and 475 library scripts using 272156virt/69388res/3492shr/65956data kb, in 550usr/80sys/635real ms.
Pass 2: analyzed script: 4 probes, 13 functions, 7 embeds, 3 globals using 431592virt/229936res/4740shr/225392data kb, in 1590usr/1710sys/3301real ms.
Pass 3: using cached /root/.systemtap/cache/1e/stap_1eac85c480fa3920fda429fabe71b526_22654.c
Pass 4: using cached /root/.systemtap/cache/1e/stap_1eac85c480fa3920fda429fabe71b526_22654.ko
Pass 5: starting run.
(coredns) with PID: 3494 sent UDP to 8.8.8.8 port 53
(coredns) with PID: 3494 sent UDP to 8.8.8.8 port 53
(coredns) with PID: 3494 sent UDP to 8.8.8.8 port 53
(coredns) with PID: 3494 sent UDP to 8.8.8.8 port 53

The script will first be compiled. Then, it will start running and printing the data of the UDP packets currently being sent.

We can also redirect the output of the script to a file using this command:

$ stap -v udp_monitor.stp > output_file

The code above adds the output of the script to a file named output_file. This way, we can analyze the output at a later point.

To test the script using ping, we can enter the ping command in a separate terminal while the script is running. We should see something similar to this in our output:

$ (ping) with PID: 16691 sent UDP to 10.199.227.16 port 53

Systemtap is a really powerful tool that has a lot of additional features such as conditions and filters we can add to our script.

4. Conclusion

In this article, we monitored which processes are sending UDP packets from a given machine.

To do this, we explored two different tools: the Linux Audit service and Systemtap. We learned how to use both of them to identify the PID of the processes sending the UDP traffic.

Basically, the Linux Audit service might be easier to use and more beginner-friendly, while Systemtap is usually more powerful and customizable.

Comments are open for 30 days after publishing a post. For any issues past this date, use the Contact form on the site.