Automatic Bug Reporting Tool (ABRT) and How to Clean /var/spool/abrt

1. Introduction

Similar to utilities like Apport and systemd-coredump, the older Automatic Bug Reporting Tool (ABRT) is a set of utilities that handles and works with core dumps. The latter are generated when irrecoverable errors like segmentation faults occur.

In this tutorial, we explore the abrtd daemon and ways to prevent it from filling the system with data. First, we get an overview of ABRT. After that, we go through its deployment steps. Next, we check out a basic usage scenario. Finally, we discuss ABRT storage maintenance options.

Notably, ABRT is specific to older RPM-based Linux distributions and is now largely replaced by systemd-coredump.

We tested the code in this tutorial on CentOS 7 with GNU Bash 4.2.46. It should work in most POSIX-compliant environments unless otherwise specified.

2. Automatic Bug Reporting Tool (ABRT)

Same as its modern counterparts, the Automatic Bug Reporting Tool (ABRT) is a service and toolkit that provides a way to collect and organize data for bug reports.

As such, it has fairly common features:

• employs own hooks to intercept crashes from different languages (C and C++, Python, Ruby, Java)
• searches system logs for strings that hint at crashes in the hardware, kernel, or display server
• detects core dumps from other services

In particular, if any issues are detected, ABRT places relevant data into files within a subdirectory in one of two main directories:

• /var/spool/abrt
• /var/tmp/abrt

Let’s see some example files we might find under a new incident subdirectory:

+-------------------+-----------------------------------------+
| Name              | Function                                |
+-------------------+-----------------------------------------+
| backtrace         | active function calls                   |
| coredump          | core dump from compiled programs        |
| count             | number of occurrences                   |
| dso_list          | dynamic libraries loaded                |
| executable        | absolute path of problematic executable |
| package           | RPM package of executable               |
| time              | UNIX timestamp of first occurrence      |
| var_log_messages  | related system log lines                |
+-------------------+-----------------------------------------+

Separately, ABRT has its own command line interface (CLI) and graphical user interface (GUI) tools for viewing and sending reports.

3. Deploy ABRT

Although some Linux installations contain ABRT, others may not include it by default. So, let’s install and configure the service and toolkit.

3.1. Installation

Two main packages provide the ABRT bundle:

• abrt-cli: headless version
• abrt-desktop: GUI version

So, let’s install abrt-cli via yum:

$ yum install abrt-cli

At this point, we should have the basic utilities.

Let’s start the service via service, since systems with systemd and systemctl usually don’t employ ABRT:

$ service start abrtd

Notably, once any of the abrt-* packages is installed, a new pattern is automatically set in /proc/sys/kernel/core_pattern:

$ cat /proc/sys/kernel/core_pattern
|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I %h

Thus, new core dumps are redirected to an appropriate ABRT hook.

3.2. Enable Core Dump Generation

However, to ensure core dumps actually do get generated, we might need to edit the /etc/security/limits.conf file:

$ cat /etc/security/limits.conf
[...]
*  soft  core  unlimited
[...]

In particular, we modify the commented line that includes the soft limit for maximum number of core dumps. After uncommenting the line, we change the value from the default of 0 to unlimited.

3.3. Process Custom Executables

Critically, if we want to enable core dump generation for executables that aren’t part of packages, we often need to change another file as well:

$ cat /etc/abrt/abrt-action-save-package-data.conf
[...]
# Process crashes in executables which do not belong to any package?
#
ProcessUnpackaged = yes
[...]

Here, we edit /etc/abrt/abrt-action-save-package-data.conf to change ProcessUnpackaged to yes.

3.4. General Configuration

In case we need to set other ABRT options, we can edit files in one of three main configuration locations:

• /etc/abrt/
• /etc/libreport/
• $HOME/.config/abrt/

For instance, let’s check part of the /etc/abrt/abrt.conf file:

$ cat /etc/abrt/abrt.conf
# Enable this if you want abrtd to auto-unpack crashdump tarballs which appear
# in this directory (for example, uploaded via ftp, scp etc).
# Note: you must ensure that whatever directory you specify here exists
# and is writable for abrtd. abrtd will not create it automatically.
#
#WatchCrashdumpArchiveDir = /var/spool/abrt-upload

# Max size for crash storage [MiB] or 0 for unlimited
#
MaxCrashReportsSize = 5000

# Specify where you want to store coredumps and all files which are needed for
# reporting. (default:/var/spool/abrt)
#
# Changing dump location could cause problems with SELinux. See man abrt_selinux(8).
#
#DumpLocation = /var/spool/abrt
[...]

Here, we see options such as the report pre-upload directory (for online reporting), maximum size of reports, as well as the expected core dump location.

4. Using ABRT

Having installed and configured the package, let’s understand how we can employ ABRT in a typical scenario.

4.1. Generate Core Dump

To begin with, we run a problematic executable and check the result:

$ ./segfault.bin
Segmentation fault (core dumped)

Now, we can check the contents of the /var/spool/abrt directory via ls:

$ ls -1 /var/spool/abrt/
ccpp-2024-01-10-10:00:01-666
last-ccpp

Thus, we can see the new ccpp-2024-01-10-10:00:01-666 directory, starting with the ccpp C and C++ prefix, continuing with the 2024-01-10-10:00:01 date and time, and ending with the 666 process ID (PID) of the offending process.

4.2. ABRT Report Files

In it, we find a number of files, some of which we already know the function of:

$  ls /var/spool/abrt/ccpp-2024-01-10-10\:00\:01-666/
abrt_version  coredump    exploitable      limits     os_release       runlevel  uuid
analyzer      count       global_pid       machineid  pid              time      var_log_messages
architecture  environ     hostname         maps       proc_pid_status  type
cgroup        event_log   kernel           open_fds   pwd              uid
cmdline       executable  last_occurrence  os_info    reason           username

Further, let’s show the sizes of all files and a summary:

$ du --total --bytes * | sort --numeric-sort
0       event_log
1       count
1       uid
4       analyzer
4       global_pid
4       hostname
4       pid
4       runlevel
4       type
5       pwd
5       username
6       abrt_version
6       architecture
10      last_occurrence
10      time
14      cmdline
18      executable
28      kernel
30      reason
36      os_release
40      uuid
125     exploitable
135     machineid
138     open_fds
187     cgroup
337     maps
393     os_info
1209    proc_pid_status
1323    limits
2016    environ
2354    var_log_messages
155648  coredump
164099  total

In this case, we use du to show the actual size in –bytes (-b) of each file, along with the –total. After that, we pipe the result to sort, so we have a more convenient way to see where the bulk of the information is – in the coredump.

In practice, the total size is around 165KB with a coredump of 155KB or around 90%.

4.3. Crash Analysis

Although we wouldn’t be able to employ them in our scenario, there are several ABRT CLI tools for analyzing core dumps:

• abrt-action-analyze-backtrace
• abrt-action-analyze-python
• abrt-action-analyze-c
• abrt-action-analyze-vmcore
• abrt-action-analyze-ccpp-local
• abrt-action-analyze-vulnerability
• abrt-action-analyze-core
• abrt-action-analyze-xorg
• abrt-action-analyze-oops

These require a package-related executable for the best results. Which one we use depends on the crash and executable type.

For example, for package-related ccpp report directories, we can use abrt-action-analyze-ccpp-local. Each command generates everything needed for a bug report, including a backtrace.

5. ABRT Size Controls

Depending on our use cases, the total quantity of dumps can become quite big.

Let’s see how we can manage their size.

5.1. Report Sizes

As we saw, a simple non-package executable generated more than 160KB of data. Thus, if we do regular development work or run a server for testing, the /var/spool/abrt/ or /var/tmp/abrt/ directories can grow a lot.

To limit this, we can use the MaxCrashReportsSize option in the /etc/abrt/abrt.conf file:

$ cat /etc/abrt/abrt.conf
[...]
# Max size for crash storage [MiB] or 0 for unlimited
#
MaxCrashReportsSize = 50

In this case, we limit the total crash report storage to 50MB. The issue is that once this limit is reached, the system automatically deletes the oldest report to make space for a newer one.

So, a single huge report may overwrite all data in the directory.

5.2. Deletion by Date

To exert finer control over the ABRT report removal, we can use a custom executable script:

$ cat abrt-delete.sh
#!/usr/bin/env bash
set -e
function startabrt()
{
  systemctl start abrtd
  systemctl start abrt-oops
}

trap startabrt EXIT

systemctl stop abrtd
systemctl stop abrt-oops
find /var/spool/abrt/ -type d -ctime +1 -exec abrt-cli rm {} \;
startabrt
$ chmod +x abrt-delete.sh

Let’s break down this script:

• set -e forces the script to exit upon any error
• function startabrt() starts the abrt and abrt-oops (kernel issues) services
• trap startabrt EXIT ensures we always leave the services started, regardless of the EXIT reason
• systemctl controls the status of the relevant services

On the other hand, the find command looks for [d]irectory -type objects within /var/spool/abrt with a creation time (-ctime) of more than one (+1) days. For any that are found, the rm subcommand of the abrt-cli tool deletes them.

Of course, we can use a scheduler like cron to automate how often this process runs.

For example, we can use the /ect/cron.daily/ directory:

$ cp abrt-delete.sh /etc/cron.daily/

Now, abrt-delete.sh checks for reports to delete and removes any matching ones every day.

6. Summary

In this article, we delved into the Automatic Bug Reporting Tool (ABRT).

In conclusion, due to the quantity of information produced by core dumps and ABRT reports, it’s good practice to know how the service works, so we can control its data storage.