How to Find Memory Leak of a Running Process in Linux

1. Introduction

Memory leaks are a common issue in long-running Linux processes, where memory that has been allocated isn’t properly released after it’s no longer needed. Over time, this can cause processes to consume excessive memory, leading to performance degradation or even system crashes.

Monitoring memory usage and identifying leaks is crucial, especially when uptime is critical, such as servers or embedded systems.

In this tutorial, we’ll explore various tools and methods to detect memory leaks in already-running processes. By the end, we’ll understand how to use these tools to monitor and troubleshoot memory leaks in real-time.

2. Understanding Memory Leaks

A program causes memory leaks when it allocates memory but doesn’t free it when it’s no longer required. This increases memory usage over time as more allocations happen without releasing old ones. Symptoms of a memory leak include gradually increasing memory usage, application slowdowns, and crashes due to resource exhaustion.

Identifying memory leaks in a running process is essential for maintaining system stability, especially in long-running applications. There are multiple tools and commands in Linux that can help track memory usage and detect these leaks efficiently. We’ll now introduce them one by one.

3. Using top and htop for Basic Monitoring

The top and htop commands both provide a high-level view of system resource usage, including memory consumption. htop is a better version of the top command that is interactive and more informative. These tools are particularly useful for quickly checking if a process is consuming too much memory over time.

We’ll focus on htop for now, which can be installed using apt:

$ sudo apt install htop

Once installed, we can run htop to monitor the system and process memory usage:

$ htop

Let’s see an example output of running the htop command:

The output of the htop command.

In the output, we can observe columns like RES (Resident Set Size) and VIRT (Virtual Memory Size), which display the memory usage of each process. Also, the MEM% column gives a direct percentage of memory usage in comparison to the entire memory. The real-time updates make it easy to see if memory usage for any particular process is continuously increasing.

By using htop, we can quickly identify suspicious memory behavior in real-time. If a process is consuming increasing amounts of memory over time without releasing it, we may be facing a memory leak.

4. Checking /proc Filesystem for Memory Leak Data

The /proc filesystem in Linux provides detailed information about running processes. Linux maps all the running processes to the filesystem, which we can access through the /proc directory. We can access memory-related data for any running process directly from the /proc directory:

$ cat /proc/[pid]/status

This command will show much information related to the process. Among these, there are several fields related to memory usage as well, such as VmRSS (Resident Set Size), VmSize (Virtual Memory Size), and VmData (Data Segment Size), which are indicators of memory usage. Monitoring these fields over time can help us detect if a process is leaking memory.

By examining these values periodically, we can track the memory usage of a running process and determine if there are unusual increases in memory allocation without corresponding releases.

5. Using pmap to Analyze Memory Usage in a Running Process

pmap gives us a detailed memory map of a running process, showing how memory is allocated and used. This command helps us inspect how memory is divided across different segments, such as the stack, heap, and shared libraries.

We can analyze the memory usage of a specific process using:

$ pmap [pid]

For example, let’s check the memory map of the process with PID 1234:

$ pmap 1234

The output might look like:

1234:   /usr/bin/someprocess
0000555555554000    1640K r-x-- /usr/bin/someprocess
0000555555713000      32K r---- /usr/bin/someprocess
000055555571b000       4K rw--- /usr/bin/someprocess
00007ffff7c00000    2048K r-x-- /lib/x86_64-linux-gnu/libc-2.31.so
00007ffff7e00000      16K r---- /lib/x86_64-linux-gnu/libc-2.31.so
00007ffff7e04000       8K rw--- /lib/x86_64-linux-gnu/libc-2.31.so
00007ffff7fff000       4K rw---   [ anon ]
00007ffffffde000     132K rw---   [ stack ]
00007ffffffff000       4K r-x--   [ anon ]

In the example above, the memory map of process 1234 is displayed, showing various memory segments like the executable, libraries, stack, and anonymous memory.

Each line contains the memory address range, size, access permissions (read, write, and execute), and the corresponding file or memory type (for example, stack, anon, or shared libraries like libc-2.31.so). The heap section isn’t explicitly shown here but can be part of anonymous memory allocations.

By analyzing the memory map with pmap, we can identify any unusually large memory segments that may point to memory leaks. Repeatedly checking the same process over time can help us spot if a particular segment, such as the heap or anonymous memory, is continuously growing, indicating a memory leak.

6. Real-Time Monitoring with smem

smem is a memory reporting tool that provides a more accurate picture of memory usage by breaking down proportional and shared memory for each process. Unlike other tools like top, which only show total memory usage, smem differentiates between memory uniquely used by a process and memory shared between multiple processes.

This is useful because shared memory (like shared libraries) can lead to over-reporting of a process’s memory usage in other tools. By showing proportional memory, smem helps us better understand the actual memory footprint of each process.

Let’s see the command to install smem:

$ sudo apt install smem

After installation, we can track the memory usage of a process in real-time:

$ smem -k

This command will output a detailed report, breaking down memory usage into categories like USS (Unique Set Size), PSS (Proportional Set Size), and RSS (Resident Set Size). Here, the -k or the –abbreviate option displays the memory usage in a human-readable form.

Let’s see an example of running smem:

$ smem -k
PID User     Command          USS     PSS     RSS
1234 root     /usr/bin/app     300K    400K    500K
5678 root     /usr/bin/db      1.5M    2.0M    2.5M

In this output:

• USS represents the memory exclusively used by this process.
• PSS represents the process’s share of memory that is shared between multiple processes.
• RSS represents the total memory used by the process, including both shared and unique memory.

smem gives us insights into how much memory a process is truly using, especially when shared memory is involved. This gives us a more realistic understanding of memory usage compared to other tools like top or htop, which don’t break down shared and unique memory separately.

As a result, smem is highly useful for identifying whether a memory leak is occurring within the unique memory space of a process or if it’s related to shared memory usage across multiple processes.

7. Automating Memory Leak Detection with memleax

memleax detects memory leaks in real time on running processes. It automatically identifies memory leaks without requiring us to restart or modify the program, making it an excellent choice for detecting leaks in live environments.

The easiest way to install memleax is using the .rpm or .deb packages, since it is only available in Arch Linux and FreeBSD repositories. Let’s see how to install it from the .deb package:

$ wget https://github.com/WuBingzheng/memleax/releases/download/v1.1.1/memleax_1.1.1-1_amd64.deb
$ sudo dpkg -i memleax_1.1.1-1_amd64.deb
$ sudo apt-get install -f

We first download the .deb package from the official repository using wget. Then, we install it using dpkg. If some dependencies are missing, we can collect them using the last command.

After installing memleax, we can now use it to track memory leaks for a running process. To attach memleax to a process, we can replace [pid] with the actual process ID:

$ sudo memleax -p [pid]

For instance, if the process ID is 1234:

$ sudo memleax -p 1234

Let’s see a sample output from memleax:

pid: 1234
malloc(128) = 0x7f8e160b3050
malloc(256) = 0x7f8e160b4090
...
memory leak detected: address = 0x7f8e160b3050, size = 128

In this example, memleax monitors the memory allocations in real time. It displays the allocation calls like malloc(128) and malloc(256) with their corresponding memory addresses.

When a memory leak is detected, memleax flags the leak, showing the memory address (0x7f8e160b3050) and the size of the leaked memory (128 bytes). Thus, it provides real-time detection without requiring a process restart.

8. Using gdb for In-Depth Memory Inspection

gdb (GNU Debugger) is a powerful debugging tool that allows us to inspect running processes in detail. When combined with the heap command found inside, we can attach gdb to a running process and analyze its memory allocations in real-time. This helps us detect any anomalies or memory leaks within the heap memory of the process.

Let’s see how to install gdb:

$ sudo apt install gdb

We can attach the newly installed gdb to a running process using the -p option with the process ID:

$ sudo gdb -p [pid]

For example, if the process ID is 1234, we can attach gdb to it with:

$ sudo gdb -p 1234

Once attached, we can use the heap command to inspect the heap memory allocations for that process:

(gdb) heap

Let’s look at an example of the output after running the heap command:

Heap chunk(s) found:
Arena 0x7ffff7dd2000:
  [0x555555767260] in use: 128 bytes
  [0x5555557672e0] in use: 256 bytes
  [0x5555557673f0] in use: 64 bytes
Total memory allocated in arena: 640 bytes

In this example, the heap command shows individual chunks of memory allocated in the heap. Each chunk is identified by its memory address (such as 0x555555767260), followed by the size of the allocated memory (such as 128 bytes or 256 bytes).

The in use label shows that the process is currently using the memory and hasn’t freed it yet. At the bottom, we can see a summary of the total memory allocated in this heap arena, which is 640 bytes in this case.

By using gdb with the heap command, we gain granular control over the inspection of memory allocations in real-time. If we observe increasing memory allocations over time without corresponding free operations, it’s a strong indicator of a memory leak.

9. Conclusion

Memory leaks can severely impact system performance, especially in long-running processes, leading to resource exhaustion and crashes. Therefore, detecting and addressing memory leaks in running processes is crucial for ensuring system stability and efficiency.

In this article, we’ve explored several tools and approaches, such as htop, /proc, pmap, smem, memleax, and gdb. Each helps us monitor and detect memory leaks in running processes in real-time. By mastering these commands, we can confidently identify and resolve memory issues in live environments.