Keep Programs Alive During a Crash or a Reboot

1. Introduction

Ensuring the resilience and continual operation of programs is a critical task for system administrators and developers alike. Whether we’re running a web server, a background data processing script, or any other service, we need a reliable way to keep these programs running.

In this article, we’ll explore various methods of keeping applications alive during crashes or reboots.

2. Using a Loop in the Shell Script

The simplest approach is to wrap the execution of our script or program within an infinite loop. This method is straightforward and requires minimal setup.

2.1. Creating a Loop

Let’s create a simple infinitive loop in Bash:

#!/bin/bash
while true; do
  our_script.sh
done

We can also add sleep after execution. That way, we won’t start immediately after a crash. That may be good if we want to wait out some transient condition that provoked the crash:

#!/bin/bash
while true; do
  our_script.sh
  sleep 10
done

2.2. Drawbacks

While effective for simple needs, this approach lacks sophistication. We can’t monitor the status of the program running. Also, we can’t stop it any other way but by sending the kill signal.

And finally, if the whole system crashes, the program won’t automatically be executed again.

3. Using Cron’s @reboot

We can supplement the previous method with the @reboot feature in cron. It ensures that our program starts on system boot. First, we need to edit our crontab:

$ crontab -e

Then, we need to add the following line:

@reboot /path/to/our/app

This will create a cron job that will execute our program on every reboot.

4. Systemd Service

systemd is the init system for modern Linux distributions and offers extensive capabilities for service management. We can use it to create a service for our program and to control its execution.

4.1. Creating a Systemd Service

Let’s create a systemd service file, for example, /etc/systemd/system/myapp.service, with the following content:

[Unit]
Description=My App

[Service]
ExecStart=/path/to/our/app
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

4.2. Enabling and Starting the Service

Now we can enable and start our service:

$ sudo systemctl enable myapp.service
$ sudo systemctl start myapp.service

After that, our app will be automatically restarted after a crash or a reboot.

4.3. Monitoring

We can also easily check the status of the created service. If we just want to know if the service is active, we can type:

$ sudo systemctl is-active myapp.service

However, if we want more extensive information, like the most recent log entries or CPU and memory usage, we can use the status command:

$ sudo systemctl status myapp.service

4.4. Stopping and Disabling Services

If we need to, we can easily stop any service we previously created:

$ sudo systemctl stop myapp.service

However, since our service is programmed to start on the system’s boot, it will still be executed after a reboot, even after manual stopping. If we want to disable it, we need to execute the disable command:

$ sudo systemctl disable myapp.service

5. Supervisord

There are multiple external tools designed to control and monitor processes. In this article, we’ll focus on Supervisord.

5.1. Setting Up Supervisord

First, we need to install supervisord if  it’s not already present:

$ sudo apt-get install supervisor

Then we must create a configuration file at “/etc/supervisord.conf”. The simplest configuration would look like this:

[program:myapp]
command=/path/to/our/app

To ensure that our program is executed on reboot and kept alive, we need to set the “autostart” and “autorestart” variables to true:

[program:myapp]
command=/path/to/our/app
autostart=true
autorestart=true

5.2. Managing Programs

Now, we can use Supervisord to execute our program:

$ sudo supervisorctl reread
$ sudo supervisorctl update
$ sudo supervisorctl start myprogram

Similarly to using systemd, our program will be automatically started on reboot and restarted on crashes.

5.3. Monitoring

We can check the status of our program by using the status command:

$ sudo supervisorctl status myprogram

6. Summary

In this article, we explored several methods to keep programs running continuously on Linux systems. While each approach has its merits, the choice depends on our needs. For system-critical services, systemd offers a robust and integrated solution, whereas supervisord provides more control for managing multiple processes.

For simple tasks, a bash loop and a cron job should suffice.