1. Overview
With the advancements in hardware and cloud technologies, a single Linux machine can host various applications and VMs. Therefore, it’s pivotal that we manage system resources like CPU usage.
We often need to use commands like top or htop to view the available/used CPU for a machine. However, if we want to view such stats for a specific process, these commands are not as straightforward.
In this tutorial, we’ll look at a few options we can use to gather per-core CPU usage of a process.
2. The top Command
Generally speaking, the top command is the most popular choice for monitoring system resources. Simple syntax and the detailed snapshot of system resources makes top a common choice for identifying the CPU usage of the machine:
$ top
top - 14:11:06 up 3 days, 19:12, 0 users, load average: 1.07, 1.18, 0.91
Tasks: 4 total, 1 running, 3 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.5 us, 4.7 sy, 0.0 ni, 90.2 id, 0.5 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem : 2015684 total, 114416 free, 1100304 used, 800964 buff/cache
KiB Swap: 1048572 total, 1010160 free, 38412 used. 813428 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 32632 1504 1412 S 0.0 0.1 0:00.05 nginx
6 nginx 20 0 33112 672 180 S 0.0 0.0 0:00.00 nginx
682 root 20 0 18132 3128 2688 S 0.0 0.2 0:00.02 bash
687 root 20 0 41060 3104 2600 R 0.0 0.2 0:00.00 topNotably, top displays the process-wise CPU usage together with the overall CPU usage. Furthermore, it also displays the utilization for all the users in the machine — for example, root and nginx from our output above.
However, we can see that this output doesn’t display per-core CPU usage. It only shows the overall used or idle CPU.
If we want to know the CPU usage of each core, we can press “1” while the top command is active:
$ top
top - 14:27:13 up 3 days, 19:28, 0 users, load average: 0.29, 0.46, 0.62
Tasks: 4 total, 1 running, 3 sleeping, 0 stopped, 0 zombie
%Cpu0 : 5.9 us, 4.9 sy, 0.0 ni, 87.8 id, 1.4 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 4.5 us, 3.8 sy, 0.0 ni, 89.9 id, 1.4 wa, 0.0 hi, 0.3 si, 0.0 st
KiB Mem : 2015684 total, 112124 free, 1100472 used, 803088 buff/cache
KiB Swap: 1048572 total, 1010160 free, 38412 used. 813268 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 32632 1504 1412 S 0.0 0.1 0:00.05 nginx
6 nginx 20 0 33112 672 180 S 0.0 0.0 0:00.00 nginx
682 root 20 0 18132 3128 2688 S 0.0 0.2 0:00.02 bash
688 root 20 0 41060 3212 2708 R 0.0 0.2 0:00.49 topNow, we can see that CPU0 utilizes 5.9%, and CPU1 utilizes 4.5% of the CPU on this two-core machine. However, as shown above, we still can’t figure out the per-core CPU usage of a specific process.
To know the core utilization, we must identify the core on which the kernel has scheduled the process threads. We can also use the taskset command to add affinity so that the kernel always schedules the process on the configured core.
3. The sar Command
The sar command is part of the sysstat package in Linux, which provides various system utilities. Additionally, the sysstat package contains a system data collector that runs as a cron job and collects all system statistics and stores them in data files. We can then view these stats with the help of the sar command.
In most machines, sysstat comes pre-installed. However, we can also install it manually using Linux package installers like Yum and Apt.
Similar to the top command, the sar command also displays the overall CPU usage by default:
$ sar -u 1 3
Linux 4.19.76-linuxkit (nginx-deployment-574b87c764-fwgw8) 08/31/20 _x86_64_ (2 CPU)
02:28:58 CPU %user %nice %system %iowait %steal %idle
02:28:59 all 3.66 0.00 3.66 0.52 0.00 92.15
02:29:00 all 4.74 0.00 5.79 1.58 0.00 87.89
02:29:01 all 3.66 0.00 4.71 0.00 0.00 91.62
Average: all 4.02 0.00 4.72 0.70 0.00 90.56Here, the arguments 1 and 3 denote that sar should print the CPU information after every one second for a maximum of three times. In order to get the per-core CPU usage, we’ll use the -P argument with the sar command:
$ sar -P ALL 1 3
Linux 4.19.76-linuxkit (nginx-deployment-574b87c764-fwgw8) 08/31/20 _x86_64_ (2 CPU)
02:36:34 CPU %user %nice %system %iowait %steal %idle
02:36:35 all 3.11 0.00 3.11 0.00 0.00 93.78
02:36:35 0 3.12 0.00 2.08 0.00 0.00 94.79
02:36:35 1 3.06 0.00 4.08 1.02 0.00 91.84
02:36:35 CPU %user %nice %system %iowait %steal %idle
02:36:36 all 3.59 0.00 3.08 0.51 0.00 92.82
02:36:36 0 3.09 0.00 3.09 0.00 0.00 93.81
02:36:36 1 4.12 0.00 2.06 0.00 0.00 93.81
02:36:36 CPU %user %nice %system %iowait %steal %idle
02:36:37 all 2.58 0.00 3.09 0.52 0.00 93.81
02:36:37 0 2.04 0.00 4.08 1.02 0.00 92.86
02:36:37 1 3.06 0.00 3.06 1.02 0.00 92.86
Average: CPU %user %nice %system %iowait %steal %idle
Average: all 3.09 0.00 3.09 0.34 0.00 93.47
Average: 0 2.75 0.00 3.09 0.34 0.00 93.81
Average: 1 3.41 0.00 3.07 0.68 0.00 92.83Here, ALL denotes that sar should display the CPU utilization for all cores. We can further filter this output to a specific core by replacing ALL with the core id like 0 or 1.
Again, taskset can be used to set core affinity to a process so that we can filter our sar output to a particular core.
4. The ps Command
Generally, we use the ps command to list the processes running on a Linux machine. We can get information like process id, threads, run command, process owner, etc. from the output of this command:
$ ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 Sep01 ? 00:00:00 nginx: master process nginx -g daemon off;
nginx 6 1 0 Sep01 ? 00:00:00 nginx: worker process
root 7 0 0 13:24 pts/0 00:00:00 bash
root 676 7 0 14:02 pts/0 00:00:00 ps -efSimilar to the above statistics, we can also see CPU% for each process with the -o argument. This argument allows us to define a custom output format from the ps command:
$ ps -eo uid,pid,tid,class,rtprio,ni,pri,psr,pcpu,comm
UID PID TID CLS RTPRIO NI PRI PSR %CPU COMMAND
0 1 1 TS - 0 19 0 0.0 nginx
101 6 6 TS - 0 19 0 0.0 nginx
0 7 7 TS - 0 19 0 0.0 bash
0 687 687 TS - 0 19 0 0.0 psHere, %CPU is the CPU usage of each thread with thread id %TID of the process. Since each thread is scheduled on a different core, we can say that this denotes the per-core CPU usage.
Although ps is the quickest way to get per-core utilization, it’s not the most reliable. It’s important to point out here that ps is a snapshot tool rather than a monitoring tool like top. Therefore, the CPU usage shown here is the average CPU time consumed by the process since it was started.
On the other hand, the output of the top command is the actual CPU consumed by the process at any given time. The top command monitors the CPU consumption and hence reports the most accurate CPU consumption of a process.
5. The mpstat Command
Similar to the sar command, the mpstat command is also a part of the sysstat package. This command reports the per-core CPU usage for all processes in the machine:
$ mpstat -P ALL 3
Linux 4.19.76-linuxkit (nginx-deployment-574b87c764-fwgw8) 09/03/20 _x86_64_ (2 CPU)
12:39:34 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
12:39:37 all 5.91 0.00 5.74 0.52 0.00 0.35 0.00 0.00 0.00 87.48
12:39:37 0 6.25 0.00 5.56 0.35 0.00 0.00 0.00 0.00 0.00 87.85
12:39:37 1 5.56 0.00 6.25 0.69 0.00 0.35 0.00 0.00 0.00 87.15
12:39:37 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
12:39:40 all 3.44 0.00 4.81 0.52 0.00 0.00 0.00 0.00 0.00 91.24
12:39:40 0 3.42 0.00 5.14 0.68 0.00 0.34 0.00 0.00 0.00 90.41
12:39:40 1 3.79 0.00 4.48 0.00 0.00 0.00 0.00 0.00 0.00 91.72
^C
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 4.67 0.00 5.27 0.52 0.00 0.17 0.00 0.00 0.00 89.37
Average: 0 4.83 0.00 5.34 0.52 0.00 0.17 0.00 0.00 0.00 89.14
Average: 1 4.67 0.00 5.36 0.35 0.00 0.17 0.00 0.00 0.00 89.45The command we used will display the overall CPU usage along with per-core usage after every 3 seconds.
This output is similar to the top and the sar command. As we’ve discussed previously, if we know the cores on which our process is running, we can easily know the CPU usage for it.
6. The dstat Command
The dstat command is part of the dstat package in Linux. Similar to other commands discussed here, dstat is a statistical command that displays system resources like CPU usage.
One major advantage that dstat has over other commands is that it allows us to input the CPU core id for which we need the CPU usage:
$ dstat -C 0,1
You did not select any stats, using -cdngy by default.
-------cpu0-usage--------------cpu1-usage------ -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| recv send| in out | int csw
5 5 89 1 0 0: 5 5 89 1 0 0| 29k 64k| 0 0 | 49B 496B| 676 5163
5 3 90 2 0 0: 5 2 93 0 0 0| 0 72k| 0 0 | 0 0 | 685 5292
3 4 93 0 0 0: 3 5 92 0 0 0| 0 12k| 0 0 | 0 0 | 855 5589The command shown above outputs the CPU usage for core 0 and core 1. On Linux machines, we can find the core id from the cpuinfo file:
$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 142
model name : Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
stepping : 10
microcode : 0xffffffff
cpu MHz : 2112.000
cache size : 8192 KB
.
.
.
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 142
model name : Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz
stepping : 10
microcode : 0xffffffff
cpu MHz : 2112.000
cache size : 8192 KB
.
.
.7. Conclusion
So, with all things considered, we can say that there’s no shortcut command to get the per-core CPU utilization for a process in Linux. And, in most cases, we need to control the cores on which a process can run to get this information. We briefly discussed the taskset command, which allows us to add CPU core affinities so that a process is scheduled on that core.
Later, we also discussed a few commands in Linux that can give us the per-core CPU utilization. Therefore, if we know which core our process is scheduled for, we can easily get the CPU usage for that core using these commands.