1. Overview
When we work with the Linux command line, we usually run Linux commands in the foreground. However, in some situations, we need to run multiple commands in the background.
In this tutorial, we’ll see how to run multiple commands as background jobs using two approaches:
- Running multiple commands as a single background job
- Running multiple commands in multiple background jobs
We’ll focus on the Bash shell in this tutorial.
2. Running Multiple Commands as a Single Job
Usually, we want to run multiple commands as a single job when the commands are related to each other.
We can start multiple commands as a single job through three steps:
- Combining the commands – We can use “;“, “&&“, or “||“ to concatenate our commands, depending on the requirement of conditional logic, for example: cmd1; cmd2 && cmd3 || cmd4
- Grouping the commands – We can group the combined commands by “{ }” or “( )“, for example: ( cmd1; cmd2 && cmd3 || cmd4 )
- Sending the command group to the background as a job – If we add the & operator after our command group, the shell will run the command group as a background job, for example: ( cmd1; cmd2 && cmd3 || cmd4 ) &
Next, let’s see an example of how to execute multiple commands as a single background job.
Let’s say we want to do an odd-even check on a number from an expensive calculation and output the result. Since the expensive calculation may take some time, we want to run the commands as a background job. So we can apply the three steps above to build the command:
$ ( echo "Calculating the number..."; sleep 8; \
NUM=$RANDOM; [ $((NUM%2)) -eq 0 ] && echo "$NUM: even" || echo "$NUM: odd" ) &In the command above, we use the sleep command to simulate the number calculation process. Now, let’s see how it is running as a job through a little demo:
As the demo shows, we started the command three times. Every time we launched the command, the command group was running as a background job. We can monitor the status of the jobs using the jobs command.
3. Controlling the Output
We’ve learned how to launch multiple commands as a single background job. If we review the demo, we’ll find all outputs from the jobs are printed to our terminal. This is because the background process started by the & operator will inherit the stdout and stderr from the shell.
In practice, we often want to redirect the output of the job to a file so that the job outputs do not mess up the current terminal. Moreover, we can check the result of the job or the log of the job execution efficiently. To achieve that, we can redirect the output of the command group to a file:
$ ( echo "Calculating the number..."; sleep 8; \
NUM=$RANDOM; [ $((NUM%2)) -eq 0 ] && echo "$NUM: even" || echo "$NUM: odd" ) > result.txt &
[1] 40030
$
[1]+ Done ( echo "Calculating the number..."; sleep 8; ... ) > result.txt
$ cat result.txt
Calculating the number...
30027: odd
Thus, the output of the job goes to the file result.txt. We only see the PID of the job and the done notification in the current terminal.
The PID of the job is essential information. Usually, we won’t want to suppress it. However, we can hide it if we don’t want to see any output after launching the command:
$ { ( echo "Calculating the number..."; sleep 8; NUM=$RANDOM; \
[ $((NUM%2)) -eq 0 ] && echo "$NUM: even" || echo "$NUM: odd" ) > result.txt & } 2>/dev/null
$
$ jobs
[1]+ Running ( echo "Calculating the number..."; sleep 8; ... ) > result.txt &
$
[1]+ Done ( echo "Calculating the number..."; sleep 8; ... ) > result.txt
$ cat result.txt
Calculating the number...
7667: odd
As the output above shows, we redirected the stderr to /dev/null to suppress the output of the “&” operator because it writes the PID information to the stderr.
If we take a closer look at the output above, we may see the done notification of the job is still printed. The done notification of a job is a feature of the shell. It cannot be controlled by IO redirection.
If we want to hide the done notification too, we have to start the job in a sub-shell by changing the “{ }” group into “( )“. However, the side effect is that we lose control of the job. Therefore, this is not recommended:
$ ( ( echo "Calculating the number..."; sleep 8; NUM=$RANDOM; \
[ $((NUM%2)) -eq 0 ] && echo "$NUM: even" || echo "$NUM: odd" ) > result.txt & )
$ jobsAs the command above shows, we don’t need to redirect the stderr if we wrap the job in a “( )” group. This is because the job is not started in the current shell, and it doesn’t use the stderr of the current shell either.
Further, we can see the jobs command outputs nothing. That is to say, the current shell doesn’t know about this job. Therefore, we can’t monitor or control the job. We don’t know when is the proper time to check our result.txt either since we don’t know if the job has been finished.
4. Running Multiple Commands as Multiple Jobs
We can also run multiple commands as different jobs to let them run in parallel.Â
To achieve that, we add the “&” operator to the command or command group we want to send to the background, for example:
cmd1 & cmd2 & (cmd3; cmd4) &In the example above, we’ll start three jobs, and they run in parallel:
- Job1: cmd1
- Job2: cmd2
- Job3: (cmd3; cmd4)
A concrete example can help us to understand it quickly:
$ date & (sleep 5; echo "cmd2 done") & (sleep 3; echo "cmd3 done") &
[1] 41012
[2] 41013
[3] 41014
Sun Sep 13 01:31:57 PM CEST 2020
$ jobs
[1] Done date
[2]- Running ( sleep 5; echo "cmd2 done" ) &
[3]+ Running ( sleep 3; echo "cmd3 done" ) &
$ cmd3 done
cmd2 done
[2]- Done ( sleep 5; echo "cmd2 done" )
[3]+ Done ( sleep 3; echo "cmd3 done" )
As the output above shows, we start three background jobs, and they run in parallel.
5. Waiting for Background Jobs’ Completion
So far, we’ve seen how to start commands as background jobs. Also, we can get the jobs’ status report using the jobs command.
Sometimes, particularly when we write shell scripts, we start a few time-consuming commands as background jobs and want to do some further calculations that rely on the result of the jobs.
In other words, we need to wait for those jobs to complete and then execute additional commands.
In this case, the jobs command isn’t that helpful, as it isn’t efficient if we, in our script, keep asking the job status and parsing the output to decide if all required jobs are finished.
Now, it’s time to introduce the wait command.
wait is a shell built-in command. We can pass the PIDs to the wait command and ask it to await those processes’ completion.
Let’s understand the wait command through an example:
$ cat await-jobs.sh
#!/bin/bash
JOB1_RESULT="/tmp/job1.result"
JOB2_RESULT="/tmp/job2.result"
rm -f $JOB1_RESULT $JOB2_RESULT
(sleep 5; echo $RANDOM > "$JOB1_RESULT") &
PID_JOB1=$!
echo "job1 started with PID $PID_JOB1"
(sleep 3; echo $RANDOM > "$JOB2_RESULT") &
PID_JOB2=$!
echo "job2 started with PID $PID_JOB2"
echo "Await two jobs' completion..."
wait $PID_JOB1 $PID_JOB2
awk 'NR==FNR{ r1=$1;printf "job1 result: %d\n", r1; next }
{ r2=$1;printf "job2 result: %d\n", r2 }
END{printf "The Sum: %d\n", r1+r2}' $JOB1_RESULT $JOB2_RESULT
The await-jobs.sh script above starts two background jobs. Each job generates a random number and writes it to a file.
To simulate the jobs that will run for a while, we’ve used the sleep command to sleep a few seconds.
It’s worthwhile to mention that the shell special variable $! will give us the PID of the last process. Here, we can get the PIDs of the background jobs using this variable.
Next, we’ve used the wait command to await both jobs’ completion.
After the two jobs are finished, they have written the results to the result files. We use an awk command to read the result files and print each result together with the sum of them.
Let’s see how the script works through a demo:
6. Conclusion
In this article, we’ve demonstrated how to execute multiple commands as a single background job. Further, we’ve discussed how to control the output of the jobs.
Moreover, we’ve addressed the technique of launching multiple commands as multiple background jobs and letting them run in parallel.
Finally, we’ve shown the usage of the wait command to await background jobs’ completion. It would be a useful technique when we write background job-related shell scripts.