1. Overview
In this tutorial, we’ll discuss how to monitor disk I/O activity in the Linux system. It’s an important task to perform while maintaining a system. Essentially, getting data back from the disk costs time. As a result, the disk I/O subsystem is considered the slowest part and can slow down the whole system.
Therefore, it’s important to check disk performance. We need to see if there’s a bottleneck and what causes it, as well as identify which process is waiting for the I/O request to finish. To this end, we’ll learn how to use tools like iostat, iotop, sar, and vmstat to check the disk I/O performance in Linux systems.
2. Install the Tools
Before we begin, we have to install the tools in our Linux system if they aren’t already available. The tools iostat, vmstat, and sar are part of the sysstat package. The iotop tool is part of the iotop package.
To install in Debian, Ubuntu, or any other derivative, we’ll run:
$ sudo apt-get install sysstat$ sudo apt-get install iotopFor RHLE, CentOS, and Fedora, we’ll run:
$ sudo dnf install sysstat$ sudo dnf install iotop3. Report Disk I/O Statistics
To start, it’s interesting to get an overview of disk I/O activity. In this case, the iostat command is handy and easy to understand. It stands for input/output statistics. It reports information about the CPU and disk device utilization.
At this time, we’ll only focus on disk I/O activities; therefore, we’ll use the -d option:
$ iostat -d
Linux 5.13.12-100.fc33.x86_64 (dhcppc5) 08/30/2021 _x86_64_ (4 CPU)
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
dm-0 4.84 48.24 35.44 0.00 15064145 11067448 0
dm-1 1.00 1.41 2.60 0.00 440364 812036 0
dm-2 0.18 5.59 0.01 0.00 1747005 2788 0
dm-3 0.08 8.02 0.58 0.00 2503710 180532 0
dm-4 0.00 0.00 0.00 0.00 1272 0 0
dm-5 0.08 8.02 0.58 0.00 2503421 181956 0
sda 3.14 63.31 38.14 0.00 19772796 11911070 0
scd0 0.00 0.00 0.00 0.00 1 0 0
zram0 7.74 9.36 21.58 0.00 2924188 6738944 0
To repeat the iostat command, we can add an interval in second:
$ iostat -d 2As we can see, iostat displays information about all device activities present in the system. We can also add the -p option to display statistics for block devices and all of their partitions:
$ iostat -d -p sda
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
sda 4.11 78.09 47.02 0.00 77566501 46701318 0
sda1 0.00 0.01 0.00 0.00 8432 6 0
sda2 4.10 78.08 47.02 0.00 77550214 46701280 0
sda3 0.00 0.01 0.00 0.00 5168 32 0
By default, the iostat command displays information about all block devices and partitions present in the system. If we only want to print information about the disks or partitions that are really active, skipping devices with zero metrics, we can add the -z option:
$ iostat -d -z 2
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
dm-0 0.50 2.00 0.00 0.00 4 0 0
sda 0.50 2.00 0.00 0.00 4 0 0
zram0 2.50 10.00 0.00 0.00 20 0 0
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
dm-0 22.50 0.00 152.00 0.00 0 304 0
dm-3 0.50 64.00 0.00 0.00 128 0 0
dm-5 0.50 64.00 0.00 0.00 128 0 0
sda 16.00 64.00 148.00 0.00 128 296 0
zram0 0.50 2.00 0.00 0.00 4 0 04. Identify the Process Behind the Bottleneck
We previously discussed how to check I/O activities, but sometimes that isn’t enough. For instance, it’s important to identify which process or thread is causing heavy I/O activities. The iotop command can help us do just that. It’s an interactive tool that provides real-time disk activities by process or thread. It’s important to note that iotop requires root privileges or NET_ADMIN capability.
Also, we have to check if the following kernel options are enabled:
$ egrep '(CONFIG_VM_EVENT_COUNTERS|TASK_IO_ACCOUNTING|CONFIG_TASKSTATS|TASK_DELAY_ACCT)' /boot/config-$(uname -r)
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_VM_EVENT_COUNTERS=y
By default, the iotop command displays a view of disk I/O by each process or thread:
$ sudo iotop
Total DISK READ : 18.27 K/s | Total DISK WRITE : 0.00 B/s
0.00 B DISK READ: 18.27 K/s | Actual DISK WRITE: 14.61 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
509 be/3 root 0.00 B/s 0.00 B/s 0.00 % 3.91 % [jbd2/dm-0-8]
247269 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.21 % [kworker/0:2-events]
2507 be/4 nwd 18.27 K/s 0.00 B/s 3.57 % 0.00 % gnome-terminal-server
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % systemd --switched-root --system --deserialize 30
2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd]
3 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_gp]
4 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_par_gp]
6 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H-events_highpri]
9 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [mm_percpu_wq]
10 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_tasks_kthre]
11 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_tasks_rude_]
12 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_tasks_trace]
13 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0]
14 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched]
We can use the -o option to show only the processes or threads actually performing I/O activity:
$ sudo iotop -o
Total DISK READ : 1864.38 K/s | Total DISK WRITE : 0.00 B/s
10.34l DISK READ: 1875.65 K/s | Actual DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
245778 be/4 nwd 608.93 K/s 0.00 B/s 62.08 % 34.04 % firefox -contentproc -childID 3 -isForBrowser -prefs~46 -appdir /usr/lib64/firefox/browser 245562 true tab
1822 be/4 nwd 105.25 K/s 0.00 B/s 0.00 % 14.13 % gnome-shell
2507 be/4 nwd 139.08 K/s 0.00 B/s 0.00 % 1.48 % gnome-terminal-server
245562 be/4 nwd 206.74 K/s 0.00 B/s 93.98 % 0.00 % firefox
246053ble_ponwd ] 18.79 K/s 0.00 B/s 24.22 % 0.00 % firefox -contentproc -childID 5 -isForBrowser -prefs~sr/lib64/firefox/browser 245562 true tab [DOM Worker]
247924 be/4 nwd 127.80 K/s 0.00 B/s 44.84 % 0.00 % vlc --started-from-file /home/nwd/Music/Mozart Allegro assai.mp4
247956 be/4 nwd 657.79 K/s 0.00 B/s 38.89 % 0.00 % dd if=/dev/zero of=/tmp/test.img bs=1G oflag=dsyncWe can add the -P option to display the list of processes without threads:
$sudo iotop -oP5. Generate Disk I/O Statistics Over a Period of Time
We previously learned how to monitor disk activities in real-time, but there are situations where we need to collect historical data for an entire week or more. The goal is to collect enough data for analysis or to plan future evolution. The simplest way to do this is by using the sar command.
By default, the sar command monitors all the system resources. In our case, we’re only interested in disk activities. Let’s use the -b option to report details about the disk activities:
$ sar -b 1
...
04:10:00 PM tps rtps wtps dtps bread/s bwrtn/s bdscd/s
04:10:01 PM 12.00 0.00 12.00 0.00 0.00 176.00 0.00
04:10:02 PM 35.00 31.00 4.00 0.00 1272.00 80.00 0.00
04:10:03 PM 0.00 0.00 0.00 0.00 0.00 0.00 0.00
04:10:04 PM 73.00 0.00 73.00 0.00 0.00 648.00 0.00
04:10:05 PM 0.00 0.00 0.00 0.00 0.00 0.00 0.00
04:10:06 PM 22.00 0.00 22.00 0.00 0.00 192.00 0.00
04:10:07 PM 1.00 0.00 1.00 0.00 0.00 8.00 0.00
04:10:08 PM 4.00 0.00 4.00 0.00 0.00 112.00 0.00
We can report activity for each block device using the -d option. We can also identify devices easily by using the -p option:
$ sar -p -d -b 1
04:26:15 PM DEV tps rkB/s wkB/s dkB/s areq-sz aqu-sz await %util
04:26:16 PM sda 2.00 128.00 52.00 0.00 90.00 0.00 0.50 0.40
04:26:16 PM sr0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
04:26:16 PM lv-root 1.00 0.00 52.00 0.00 52.00 0.00 1.00 0.20
04:26:16 PM lv-swap 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
04:26:16 PM zram0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
04:26:16 PM lv-home 1.00 128.00 0.00 0.00 128.00 0.00 1.00 0.20
...
It’s possible to save the output of the sar command in a file with the -o option. Note that the file is in a binary format:
$ sar 2 5 -o /tmp/data_io > /dev/null 2>&1
Now we’ll use the -f option to read the report generated by the sar command saved in the file:
$ sar -f /tmp/data_io6. Measure Disk I/O Usage With vmstat
Another helpful tool is vmstat, which gives an overall view of what’s happening in the system. For example, we can use the vmstat command to report information about processes, memory, and CPU and show disk I/O activity.
To retrieve disk I/O statistics from the system using vmstat, we invoke:
$ vmstat [-D] [-d] [-p partition] [interval [count]]We’ll use the -d option to display individual disk statistics:
$ vmstat -d 1
isk- ------------reads------------ ------------writes----------- -----IO------
total merged sectors ms total merged sectors ms cur sec
sda 2659118 1531393 166084689 49623955 1725402 4672072 104580917 70770454 0 25194
sr0 24187 28 5886957 1315209 0 0 0 0 0 966
dm-0 2178560 0 114338954 50785360 3433060 0 81458000 79180110 0 21302
dm-1 1605211 0 12845592 24735782 2946339 0 23570712 947720915 0 5866
zram0 2609461 0 20875688 26812 4958733 0 39669864 57039 0 511
dm-2 312586 0 21187074 3814922 2852 0 49288 76928 0 674
dm-3 88452 0 17673344 640177 4957 0 526864 104362 0 537
...
Then we’ll introduce the -p option to obtain detailed performance statistics about a partition:
$ vmstat -p /dev/sda2 1
sda2 reads read sectors writes requested writes
2663617 166204812 1725915 104595152
2663617 166204812 1725915 104595152
2663617 166204812 1725915 104595152
2663618 166205068 1725915 104595152
2663618 166205068 1725917 104595216
...7. Conclusion
In this article, we learned how to use the iostat, vmstat, and sar commands to check disk I/O performance. We also discussed how to check disk read and write activity by process using the iotop command.