1. Introduction

As with any other major operating system (OS), under Linux, processes have a process ID (PID). In essence, the PID is a number that identifies the process to the kernel. However, there are scenarios in which this number isn’t immediately obvious or available. In fact, they most often involve network processes.

In this tutorial, we discuss processes that seem to have no PID. First, we briefly describe network processes. Next, we create three types of listening processes and connect to them. After that, we go over ways to get the PID of a network process. Finally, we explore an alternative path for identifying a process.

We tested the code in this tutorial on Debian 11 (Bullseye) with GNU Bash 5.1.4. It should work in most POSIX-compliant environments.

2. Network Processes

Network connections require a process:

Yet, sometimes we might not see the PID of that process even with standard tools like netstat and lsof. In fact, network processes form the general reason for an obfuscated link between processes and their PID numbers.

Importantly, a stderr warning precedes the output of both netstat and lsof when running them as a regular user:

$ whoami
$ netstat
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
$ lsof
lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
Output information may be incomplete.

After using whoami to verify the shell runs under the regular user baeldung, we run both netstat and lsof to see the warnings.

Now, let’s create different types of connections and see how we can identify the processes behind them.

3. TCP/IP Server

Let’s start with a basic TCP/IP connection. To begin with, we create a server and client. After that, we attempt to identify the processes behind them.

3.1. Create Client and Server as root

First, we create a small server as the root user with netcat (nc):

$ nc -nvlp 6660
listening on [any] 6660 ...

While preventing DNS name resolutions with -n, we [-l]isten for connections on [-p]ort 6660 of any interface, ensuring sufficient [-v]erbosity.

After that, from another terminal, we connect to our server as a client:

$ nc localhost 6660

If we now enter any text and press Return, the listening side would print it out after a line like connect to [] from (UNKNOWN) [] 60667. At this point, we have client and server TCP/IP network processes.

3.2. Identify TCP/IP Processes

At this point, we check for our TCP/IP server and client as the baeldung regular user with both netstat and lsof:

$ whoami
$ netstat
Active Internet connections (only servers)
Proto  Recv-Q  Send-Q  Local Address    Foreign Address  State        PID/Program name
tcp        19       0  localhost:60667  localhost:6660   ESTABLISHED  -
$ lsof -i :6660

To do so, we leverage the -p flag of netstat to include the PID/Program name column, while the -i flag of lsof filters by port alone.

Notably, as a regular user, we don’t see data in the PID/Program name column of netstat, and the lsof command doesn’t provide any output at all.

Now, let’s try as root:

$ whoami
$ netstat -p
Active Internet connections (only servers)
Proto  Recv-Q  Send-Q  Local Address    Foreign Address  State        PID/Program name
tcp        19       0  localhost:60667  localhost:6660   ESTABLISHED  10666/nc
$ lsof -i :6660
nc       10666 root    4u  IPv4 4656028      0t0  TCP localhost:6660->localhost:60667 (ESTABLISHED)
nc       10666 root    3u  IPv4 4653760      0t0  TCP localhost:60667->localhost:6660 (ESTABLISHED)

In each case, we see the server PID is 10666, but only as a superuser.

4. Named UNIX Socket Connection

The netcat utility has a traditional version nc.traditional, but also a netcat-openbsd (nc.openbsd) version.

First, let’s install the latter with apt:

$ apt install netcat-openbsd

This should ensure we run netcat-openbsd when executing nc or netcat. Alternatively, we can specify the desired binary directly via nc.openbsd or nc.traditional.

4.1. Create Connection

By utilizing the -U option of nc.openbsd, we can create a named UNIX socket and [-l]isten for connections without resolving [-n]ames:

$ nc.openbsd -nvlU /home/baeldung/unix.socket
Bound on /home/baeldung/unix.socket
Listening on /home/baeldung/unix.socket

Now, we use the -U option with the same socket path without [-l]istening in another terminal:

$ nc.openbsd -U /home/baeldung/unix.socket

Once connected, the server side tells us Connection received on /home/baeldung/unix.socket. Again, after typing out some text and pressing Return, we see the message in both the client and server, our new network processes.

4.2. Identify Named UNIX Socket Processes

Once again, we use netstat and lsof:

$ whoami
$ netstat -ap
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     7660010  -                    /home/baeldung/unix.socket
unix  3      [ ]         STREAM     CONNECTED     7660011  -                    /home/baeldung/unix.socket
$ lsof +E | grep unix.socket

Since this time we check for named sockets, we add the -a switch to netstat, showing all network connections. On the other hand, +E includes named UNIX sockets in the output of lsof, which we filter through grep.

As with TCP/IP, we either don’t get the socket at all (lsof) or we don’t see the information about any associated processes (netstat).

Yet, checking with root, we can see the information:

$ whoami
$ netstat -ap
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
unix  2      [ ACC ]     STREAM     LISTENING     7660010  6671/nc            /home/baeldung/unix.socket
unix  3      [ ]         STREAM     CONNECTED     7660011  6672/nc            /home/baeldung/unix.socket
$ lsof +E | grep unix.socket
nc        6667                                root    3u     unix 0x00000000054403ab      0t0    7660010 /home/baeldung/unix.socket type=STREAM
nc        6668                                root    4u     unix 0x000000006111cdfc      0t0    7660011 /home/baeldung/unix.socket type=STREAM ->INO=4812980 236459,nc,3u

There is an extra possibility here that the socket or process is owned by the kernel itself.

5. Network FileSystem (NFS) Server

The Network FileSystem (NFS) is an integrated way for a Linux system to seamlessly and securely serve and consume files over a network. As such, NFS is part of the Linux kernel.

Although most Linux distributions ship with the packages preinstalled, we can manually install the server and client with, e.g., apt:

$ apt install nfs-kernel-server nfs-common

Now, we can create an NFS connection.

5.1. Create Client and Server

Similar to the root directory of web servers, let’s now create an export directory and open its permissions with chmod:

$ mkdir /mnt/nfspoint
$ chmod 777 /mnt/nfspoint

Next, we designate the directory as such in the /etc/exports file by adding a line to it:

$ cat /etc/exports
/home/baeldung/nfspoint *(rw,sync)

In this case, we allow access to * any client, providing [r]ead and [w]rite permissions and ensuring the data is in sync.

Finally, we use exportfs to make the share available and restart the nfs-kernel-server:

$ exportfs -a
$ systemctl restart nfs-kernel-server

The -a switch ensures we handle all exports.

At this point, we can mount the filesystem from any machine as long as the firewall allows it. To make things simple, we use localhost, i.e.,

$ mkdir /home/baeldung/nfspoint
$ sudo mount localhost:/mnt/nfs /home/baeldung/nfspoint
$ cd /home/baeldung/nfspoint

After these commands, we have the /home/baeldung/nfspoint NFS mount point along with its supporting processes.

5.2. Identify NFS Processes

Notably, as part of the kernel modules, NFS doesn’t require much setup but also avoids the usual process forking, resulting in no actual visible processes for an NFS connection even as root:

$ whoami
$ netstat -ap
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0   *               LISTEN      -

This is important since the kernel doesn’t spawn regular processes for NFS connections. Yet, we might be able to identify another association.

6. inode Data

For some network processes, extracting an associated inode can be helpful. Still, in most cases, we need superuser privileges to get data from the inode.

How we extract process information from its inode can depend on the filesystem.

For example, the find command with its -inum option is fairly universal:

$ find / -inum <INODE_NUMBER>

On the other hand, sometimes this might not get the information we’re after. In such cases, we can use alternative commands:

  • btrfs-inspect-internal inode-resolve <INODE_NUMBER> <PATH> for a BTRFS filesystem
  • xfs_db xfs_db -c ‘blockget -n -i <INODE_NUMBER>’ <BLOCK_DEVICE> for an XFS filesystem
  • debugfs -R ‘ncheck <INODE_NUMBER&gt’ <BLOCK_DEVICE> for ext* filesystems

In each case, we might be able to use the inode number to get further information about a given network connection.

7. Summary

In this article, we went over network processes and why they might seem to lack a PID.

In conclusion, while most processes have an ID regardless of different command outputs, some are actually internal kernel structures that don’t need a PID.

Comments are closed on this article!