Authors Top

If you have a few years of experience in the Linux ecosystem, and you’re interested in sharing that experience with the community, have a look at our Contribution Guidelines.

1. Overview

One of the tasks of the operating system is distributing the workload among processors. The scheduler ensures the optimal assignment of CPUs to tasks. However, sometimes we want to leave certain processors out of the scheduler’s reach. This way, we can create a set of isolated CPUs and transfer selected tasks to them.

In this tutorial, we’ll learn how to isolate CPUs and manage tasks assigned to them.

2. The isolcpus Kernel Parameter

With the isolcpus parameter, we can isolate specific CPUs from the kernel scheduler during the boot time. So, we should provide the value of isolcpus via GRUB. Notice that the advantage of this way is its simplicity.

2.1. One-Time Setting

To temporarily set this parameter, we need to edit the GRUB entry. So, let’s restart the computer and jump to the GRUB menu:

Next, we’re going to press e to edit the Ubuntu entry. Then, let’s add isolcpus=2-3 at the end of the linux line. In this way, we’re going to remove the third and fourth CPUs from the normal scheduling:

After the system’s been restarted, let’s check that our changes applied:

$ cat /sys/devices/system/cpu/isolated
2-3

We should be aware that the GRUB menu may be hidden. Then we need to change the GRUB configuration to make it accessible.

2.2. Permanent Change

Now let’s set isolcpus permanently by editing the grub configuration file:

$ sudo nano /etc/default/grub

Then, let’s provide isolcpus in the GRUB_CMDLINE_LINUX entry:

GRUB_CMDLINE_LINUX="isolcpus=2-3"

This time we’re going to reserve the third and fourth CPUs. Finally, let’s update the grub:

$ sudo update-grub

So, the change will come into effect after the next boot.

3. Demonstrating the Isolation

Now let’s assume that we’ve isolated the third and fourth CPUs. So, we’re going to start CPU-consuming tasks and check if these CPUs really stay idle. Hence, let’s use the sar command to collect the CPU statistics:

$ sar -P 0-3 1 70

So, we’ll collect data for CPUs 0 through 3 by 70 seconds with a one-second interval. Moreover, we run sar inside the graphical kSar interface to get charts right away. Now, let’s assume that sar is running and dispatch the CPU hogs of stress-ng:

$ stress-ng --cpu 8 --timeout 60s --metrics-brief

Finally, let’s examine the results:

So, the effect is striking. In spite of a huge workload, only CPUs 0 and 1 are loaded. Next, we can find out that the isolated processor really remained idle.

4. How to Use the Isolated Processors

Using the isolcpus parameter, we only ask the kernel not to use the indicated processors. However, we need to set the processor affinity of the process ourselves.

As an example, let’s first dispatch four CPU stressors:

$ stress-ng --cpu 4 --timeout 600s --metrics-brief

Next, let’s examine the tasks’ CPU affinity with the ps command:

$ ps -eo psr,command,pid | tr -s " " | grep "stress-ng-cpu"
 1 stress-ng-cpu [run] 4739
 0 stress-ng-cpu [run] 4740
 1 stress-ng-cpu [run] 4741
 0 stress-ng-cpu [run] 4742

So we can learn from the psr column that all tasks run on a non-isolated CPU as expected. Now, let’s move them to the isolated CPUs with help of the taskset command. Of course, we could do that by hand, but for many tasks, we’ll rather parse the ps output and use xargs:

$ ps -eo command,pid | tr -s " " | grep "^stress-ng-cpu" | cut -d " " -f 3 | xargs -n1 taskset -cp 2-3
pid 4739's current affinity list: 0,1
pid 4739's new affinity list: 2,3
# ...

Finally, let’s check the effect:

$ ps -eo psr,command,pid | tr -s " " | grep "stress-ng-cpu"
 3 stress-ng-cpu [run] 4739
 2 stress-ng-cpu [run] 4740
 3 stress-ng-cpu [run] 4741
 2 stress-ng-cpu [run] 4742

As we can see from the ps output, all stressors tasks are running now on previously unused CPUs 3 and 4.

5. Creating Processor Shielding With cset

cset is a Python application to create, separate, and manage sets of CPUs. Let’s keep in mind that cset uses the cpuset pseudo-filesystem under the hood. Moreover, the command provides a ready-to-use system of CPU sets. We can activate them by the shield super command.
Afterwards, we play with three subsets – root, user, and system. The root one contains all CPUs and makes them available for kernel threads. Then, in the user CPU set, we should run all processes we want to isolate. Finally, the system set runs all regular system tasks.

When we issue the shield command, almost all tasks from the initial root set are transferred to the system set. Then, thanks to the inheritance of processor affinity, all spawned processes stay within this set. On the other hand, we can explicitly start tasks in the user CPU set or move them from the system set. Eventually, the root set hosts only kernel tasks. So, we’ve achieved the isolation or shielding between user’s and system‘s tasks.

As opposed to the isolcpus method, cset allows the task scheduler to balance workload independently within sets.

To use cset, we should install the cpuset package. On Ubuntu or Debian distribution, we can use apt-get:

$ sudo apt-get update -y
$ sudo apt-get install -y cpuset

Moreover, as the command provides a bunch of subcommands and options, let’s learn how to obtain the help information:

$ cset help                     # for information about all subcommands
$ cset help <subcommand>        # for subcommand's details
$ cset help <subcommand> --help # for subcommand's options

5.1. Creating the Shield

Now let’s use the shield command to create a set of two processors, the third and fourth ones:

$ sudo cset shield --cpu 2,3
cset: --> activating shielding:
cset: moving 878 tasks from root into system cpuset...
[==================================================]%
cset: "system" cpuset of CPUSPEC(0-1) with 878 tasks running
cset: "user" cpuset of CPUSPEC(2-3) with 0 tasks running

Afterwards, let’s list all sets with the set command to cset:

$ cset set -l
cset: 
         Name       CPUs-X    MEMs-X Tasks Subs Path
 ------------ ---------- - ------- - ----- ---- ----------
         root        0-3 y       0 y   102    2 /
         user        2-3 y       0 n     0    0 /user
       system        0-1 y       0 n   873    0 /system

So we can recognize all the abovementioned CPU sets. From the CPUs and MEMs columns we can learn about CPUs and memory nodes used by the set. The X flag informs if the respective resource is exclusive to a given set and describes between-sets relations.

Now, let’s take a look at the number of processes using each set. So, the user set is not populated altogether, as we haven’t added any process to it yet. Next, the root set keeps 120 special tasks – kernel threads. Finally, all normal 873 processes have been moved to the system set.

6. Querying the Set’s Content

Now let’s learn commands to provide information about the processes in the shield. First, let’s use the combination of s and v options to list all shielded processes:

$ cset shield -s -v
cset: "user" cpuset of CPUSPEC(2-3) with 0 tasks running
cset: done

So, at this moment no one process uses the shield. Next, let’s list all processes outside the shield, by means of u and v options:

$ cset shield -u -v
cset: "system" cpuset of CPUSPEC(0-1) with 596 tasks running
   USER       PID  PPID SPPr TASK NAME
   -------- ----- ----- ---- ---------
   root         1     0 Soth [systemd]
   root       286     1 Soth [systemd-journal]
   root       346     1 Soth [systemd-udevd]
   systemd-   730     1 Soth [systemd-oomd]
# ...
   joe       5770  5769 Soth bash 
   joe       5816  5770 Roth /usr/bin/python3 /usr/bin/cset shield -u -v 
cset: done

In this way, we’ve displayed the content of the system CPU set. Now, let’s explain the meaning of the SPPr column. It stands for the task’s State, Policy, and Priority. Consequently, cset is running while bash is sleeping. Moreover, oth means the default, OTHER scheduling policy for the corresponding tasks, which in turn makes their priorities irrelevant.

Finally, if we skip the verbose v option, we’re going to obtain a brief summary:

$ cset shield -s
cset: "user" cpuset of CPUSPEC(2-3) with 0 tasks running
cset: done
$ cset shield -u
cset: "system" cpuset of CPUSPEC(0-1) with 595 tasks running
cset: done

7. Managing Processes in the Shield

With the shield command, we can both start tasks straight in the user CPU set or move already running processes into or out of this set. Thus, let’s execute a task with the shield and exec options:

$ cset shield --shield --exec <command> -- <options to command>

Next, we can move already running processes with help of the pid option:

$ cset shield --shield --pid <process PID>

With this option, we can pass more PIDs in a form of a comma-separated list:

$ cset shield --shield --pid <PID1,PID2,...>

Moreover, pid accepts a range of PIDs limited by initial PIDi and final PIDf, inclusively:

$ cset shield --shield --exec --pid <PIDi-PIDf>

Working with range, we don’t need to bother if some PIDs within it don’t exist, as it’s not regarded as an error. Additionally, we can combine lists and ranges in one call to cset.

Finally, let’s use the unshield option to shield if we want to remove tasks from the user set. For that, we need to arrange processes’ PIDs in a list or range in the same way as for shield:

$ cset shield --unshield --pid <single PID, list or range>

7.1. Starting Processes in the Shield

Now let’s start and shield the stressor processes:

$ cset shield --shield --exec stress-ng -- --cpu 2 --timeout 60s --metrics-brief

We’ve used the double dash after the stress-ng command to separate this command’s options from those passed to cset. Next, let’s check the shield:

$ cset shield -s -v
cset: "user" cpuset of CPUSPEC(2-3) with 4 tasks running
   USER       PID  PPID SPPr TASK NAME
   -------- ----- ----- ---- ---------
   joe       4953  4952 Soth [bash]
   joe       6118  4953 Soth [stress-ng]
   joe       6119  6118 Roth [stress-ng]
   joe       6120  6118 Roth [stress-ng]

7.2. Moving the Running Process to the Shield

We can assign a running task to the shield, so let’s do that with the current shell. Consequently, the user set will accept all processes which start in the corresponding terminal:

$ sudo cset shield --shield --pid $$
cset: --> shielding following pidspec: 11418 
cset: done

So, we’ve used the shield option to the shield command to assign the task of PID passed with the pid option. Then, let’s check the shield’s content:

$ cset shield -s -v
cset: "user" cpuset of CPUSPEC(2-3) with 2 tasks running
   USER       PID  PPID SPPr TASK NAME
   -------- ----- ----- ---- ---------
   joe       4953  4952 Soth bash 
   joe       4977  4953 Roth /usr/bin/python3 /usr/bin/cset shield -s -v 
cset: done

Now we have only one running process – the cset command itself. Thus, let’s start two stressors in the background:

$ stress-ng --cpu 2 --timeout 60s --metrics-brief &

And check the shield again:

$ cset shield -s -v
cset: "user" cpuset of CPUSPEC(2-3) with 5 tasks running
   USER       PID  PPID SPPr TASK NAME
   -------- ----- ----- ---- ---------
   joe       4953  4952 Soth bash 
   joe       5275  4953 Soth stress-ng --cpu 2 --timeout 60s --metrics-brief 
   joe       5276  5275 Roth stress-ng-cpu [run]                             
   joe       5277  5275 Roth stress-ng-cpu [run]                             
   joe       5278  4953 Roth /usr/bin/python3 /usr/bin/cset shield -s -v 
cset: done

8. Conclusion

In this tutorial, we learned how to manage CPU access for a process. First, we used the isolcpus kernel parameter to remove CPUs from the outreach of the kernel scheduler. Then, we went through the example of how this setup worked. Then, we bound processes to isolated CPUs with the taskset command.

Next, we focused on cset, a mighty command to create CPUs sets and to host tasks inside. We used its predefined schema, shield, to create a set of CPUs shielded from normal system tasks. Finally, we found out how to run processes in the isolated user set and how to move already running tasks from or to it.

Authors Bottom

If you have a few years of experience in the Linux ecosystem, and you’re interested in sharing that experience with the community, have a look at our Contribution Guidelines.

guest
0 Comments
Inline Feedbacks
View all comments