In this tutorial, we’ll make an encrypted backup of a local unencrypted directory to a remote server via rsync over ssh. We’ll also see how to keep our encrypted backup up-to-date by synchronizing it with local changes.
Our primary goal is to ensure that a file-based encryption tool encrypts on the fly all files sent by rsync. The encryption must be robust and fast, with minimal load on the CPU and no demand for extra space on the local disk.
In addition, we want a way to restore individual files from the encrypted backup without having to download and recover the entire backup.
One advantage of file-based encryption, instead of disk encryption, is that encrypted files can be synchronized efficiently using standard tools like rsync or Dropbox-like synchronization utilities.
Let’s look in more detail at our goals:
- The local directory to be backed up is not encrypted.
- Instead, an encrypted temporary copy of our local directory is exposed to rsync via a mountable FUSE virtual filesystem.
- The encrypted files are automatically updated on the fly and take up no disk space.
- An attacker with access to the server and data traffic sees only encrypted data, with no way to decrypt it.
- We should be able to transfer hundreds or thousands of encrypted gigabytes, even with big files, to the remote server via an ssh connection.
- Excluding particular files from the encrypted backup must be possible.
- Recovering individually encrypted files without the need to re-download the entire archive is a must.
- rsync needs to be as resistant as possible to network or manual interruptions, resuming synchronization from where it was interrupted.
- Limiting the maximum speed of rsync so as not to saturate the available bandwidth is desirable.
Let’s note that root permissions aren’t required, except if the backup includes files not belonging to the current user.
3. Script to Perform an Encrypted Backup
Before proceeding further, let’s ensure that, for the proper functioning of our scripts, rsync connects via ssh without requiring a password. For this aim, it’s enough to copy the local SSH key to the remote server.
That said, we can make encrypted backups based on our goals via rsync with gocryptfs in reverse mode. However, as conceptually simple as this is, there are other aspects to keep in mind in real-world use cases. For instance, our script exit codes must respect Bash standards. In fact, we have a zero value in case of success and a non-zero value in case of failure. This allows for proper integration with other scripts.
Let’s quickly look at the complete code of our Bash script. Through the use of comments, it’s divided into sections that we’ll later analyze:
#!/bin/bash #=============================================================== # Constants #=============================================================== LOOP=true # rsync loop, helpful with not stable internet connections, it can be true or false [email protected] # replace x.x.x.x with the remote server's IP or host name BWLIMIT=0 # maximum transfer rate (kbytes/s), 0 means no limit echo "rsync --bwlimit setted to $BWLIMIT kbytes/s (0 means no limit)" ENCVIRDIR=/home/francesco/toBeCopied # encrypted virtual read-only directory LOCALDATA="/media/francesco/106bfc11-23d5-49c1-8c10-953cbb082a14/test" # local data to be backed up REMOTEDIR=/mnt/blockstorage/testBackup # dir in the remote server ENCBACKUP=$SERVER:$REMOTEDIR # rsync destination dir #=============================================================== # Set a trap for CTRL+C to properly exit #=============================================================== trap "fusermount -u $ENCVIRDIR; rm -fR $ENCVIRDIR; echo CTRL+C Pressed!; read -p 'Press Enter to exit...'; exit 1;" SIGINT SIGTERM #=============================================================== # Initial checks #=============================================================== # rsync must be installed if [ -z "$(which rsync)" ]; then echo "Error: rsync is not installed" read -p "Press Enter to exit..." exit 1 fi # gocryptfs must be installed if [ -z "$(which gocryptfs)" ]; then echo "Error: gocryptfs is not installed" read -p "Press Enter to exit..." exit 1 fi #=============================================================== # Mount and rsync virtual encrypted directory #=============================================================== mkdir -p $ENCVIRDIR if find "$ENCVIRDIR" -mindepth 1 -maxdepth 1 | read; then echo "The encrypted virtual directory $ENCVIRDIR must be empty!" exit 1 fi if find "$LOCALDATA" -mindepth 1 -maxdepth 1 | read; then echo "The unencrypted directory $LOCALDATA contains local data to be backed up..." else echo "The unencrypted directory $LOCALDATA cannot be empty, it must contain local data to be backed up..." exit 1 fi if test -f "$LOCALDATA/.gocryptfs.reverse.conf"; then echo "The unencrypted directory $LOCALDATA is already initialized for gocryptfs usage." else echo "Initializing read-only encrypted view of the unencrypted directory $LOCALDATA," echo "without encrypting file names and symlink (to allow the recovery of individual files)." gocryptfs -reverse -init -plaintextnames $LOCALDATA fi # mount read-only encrypted virtual copy of unencrypted local data: if gocryptfs -ro -one-file-system -reverse -exclude-wildcard img2.jpg $LOCALDATA $ENCVIRDIR; then echo "gocryptfs succeeded -> the decrypted dir $LOCALDATA is virtually encrypted in $ENCVIRDIR" else echo "gocryptfs failed" read -p "Press Enter to exit..." exit 1 fi while [ "true" ] do # rsync local encrypted virtual copy of data to destination dir: if rsync --bwlimit=$BWLIMIT -P -a -z --delete $ENCVIRDIR/ $ENCBACKUP; then echo "rsync succeeded -> a full encrypted copy of $LOCALDATA is ready in $ENCBACKUP" break else if ! $LOOP; then echo "rsync failed" fusermount -u $ENCVIRDIR read -p "Press Enter to exit..." exit 1 fi fi if ! $LOOP; then break fi done # unmount encrypted virtual copy of local data : fusermount -u $ENCVIRDIR # remove encrypted virtual directory rm -fR $ENCVIRDIR read -p "Press Enter to exit..." exit 0
Although the code should be overall self-explanatory, let’s now take a close look at its various sections.
3.1. Constants, Trap for CTRL+C, and Initial Checks
We inserted constants to be customized, related to directory paths, the server’s IP or domain, and the maximum transfer rate that defaults to no limit. Our comments in the code specify the meaning of each constant.
The boolean constant $LOOP deserves a separate note. If set to true, rsync runs indefinitely until its exit code indicates the synchronization completion. This is a handy little trick to ensure that the backup reaches the end even on unstable connections, restarting rsync if terminated by a failure in Internet connectivity.
When a Bash script executes external commands, such as rsync or gocryptfs, each CTRL+C pressed sends a SIGINT to that running process without terminating the entire script. This is especially problematic if the command is inserted inside an infinite loop, as we’ll not be able to terminate the script. So, to ensure we properly exit our script by stopping rsync and umounting the gocryptfs‘ FUSE file system, we had to create a “trap” for CTRL+C. In fact, we used the Bash built-in trap command to specify the code to be executed when the user presses CTRL+C.
Finally, all the initial checks terminate the script in case of problems, printing on the screen what’s wrong. Aside from verifying the presence of rsync and gocryptfs, the subsequent checks on the paths are critical to avoid disasters. For example, if an unexpected condition causes synchronization of an empty local directory with our remote backup, then rsync would fully erase it with complete data loss. So, the more checks our script does, the better.
3.2. Parameters of gocryptfs
We chose suitable parameters for our goals:
- -reverse → shows a read-only encrypted view of an unencrypted directory inside a FUSE virtual file system
- -init → instructs gocryptfs to create the gocryptfs.conf file, containing the encrypted master key and its hidden copy .gocryptfs.reverse.conf
- -plaintextnames → keeps the file names in plain text, which is helpful for quickly restoring individual files from a backup
- -ro → read-only mounting, it’s already implied by the -reverse option, we added it for clarity only
- -one-file-system → doesn’t cross filesystem boundaries, so mount points will appear as empty directories
- -exclude-wildcard → excludes files from the encrypted view, it uses .gitignore syntax and matches files anywhere
Moreover, we added a check for the presence of .gocryptfs.reverse.conf to distinguish whether initialization via -init has already been done or not.
3.3. Parameters of rsync
We chose the most suitable parameters for transferring data to a remote server:
- –bwlimit → sets the maximum bandwidth to be used in kbytes/sec (0 means no limit)
- -P → keeps partially transferred files, necessary for resuming interrupted uploads, and shows progress during transfer
- -a → enables recursion and preserves symlinks, permissions, modification times, groups, owners, device files, and special files
- -z → compresses file data during the transfer
- –delete → delete extraneous files from destination directories
Let’s note that in success, failure, or manual termination via CTRL+C, we always ensure that, after the rsync execution, the gocryptfs‘ FUSE file system is unmounted by fusermount -u.
4. Script to Restore From an Encrypted Backup
Restoring from an encrypted backup requires rsync with gocryptfs in forward mode. Let’s look at the Bash script right away. Again the code is self-explanatory:
#!/bin/bash #=============================================================== # Constants #=============================================================== LOOP=true # rsync loop, helpful with not stable internet connections, it can be true or false [email protected] # replace x.x.x.x with the remote server's IP or host name BWLIMIT=0 # maximum transfer rate (kbytes/s), 0 means no limit echo "rsync --bwlimit setted to $BWLIMIT kbytes/s (0 means no limit)" DECVIRDIR=/home/francesco/toBeRestoreVirtualDecrypted # decrypted read-only virtual directory LOCALDATA=/home/francesco/toBeRestoreEncrypted # local copy of encrypted remote backup REMOTEDIR=/mnt/blockstorage/testBackup # dir in the remote server ENCBACKUP=$SERVER:$REMOTEDIR # encrypted source directory #=============================================================== # Set a trap for CTRL+C to properly exit #=============================================================== trap "echo CTRL+C Pressed!; read -p 'Press Enter to exit...'; exit 1;" SIGINT SIGTERM #=============================================================== # Initial checks #=============================================================== # rsync must be installed if [ -z "$(which rsync)" ]; then echo "Error: rsync is not installed" read -p "Press Enter to exit..." exit 1 fi # gocryptfs must be installed if [ -z "$(which gocryptfs)" ]; then echo "Error: gocryptfs is not installed" read -p "Press Enter to exit..." exit 1 fi #=============================================================== # rsync virtual directory and mount it #=============================================================== mkdir -p $DECVIRDIR mkdir -p $LOCALDATA if find "$DECVIRDIR" -mindepth 1 -maxdepth 1 | read; then echo "The decrypted virtual directory $DECVIRDIR must be empty!" echo "You can umount it with:" echo "fusermount -u $DECVIRDIR" fi while [ "true" ] do # rsync remote encrypted copy of data to local copy: if rsync --bwlimit=$BWLIMIT -P -a -z --delete $ENCBACKUP/ $LOCALDATA; then echo "rsync succeeded -> a full copy of $ENCBACKUP encrypted data is ready in $LOCALDATA" break else if ! $LOOP; then echo "rsync failed" read -p "Press Enter to exit..." exit 1 fi fi if ! $LOOP; then break fi done if ! test -f $LOCALDATA/gocryptfs.conf; then echo "The encrypted local directory $LOCALDATA cannot decrypted because gocryptfs.conf is missing!" exit 1; fi # mount read-only decrypted virtual copy of encrypted data: if gocryptfs -ro $LOCALDATA $DECVIRDIR; then echo "gocryptfs succeeded -> the encrypted dir $LOCALDATA is virtually decrypted in $DECVIRDIR" echo "You can now rsync the virtual dir $DECVIRDIR in a safe place" echo "When you done, remember to umount the virtual dir with:" echo "fusermount -u $DECVIRDIR" else echo "gocryptfs failed" read -p "Press Enter to exit..." exit 1 fi read -p "Press Enter to exit..." exit 0
This time we’ll not comment on the code because everything we’ve seen previously is sufficient to understand it.
5. Usage Examples
In the following examples, we assume that sync.sh is the script to back up and syncRestore.sh is the script to restore from the backup. These two scripts are identical to those above, except for the actual server IP instead of x.x.x.x. The directory to be backed up has the following structure:
$ tree '/media/francesco/106bfc11-23d5-49c1-8c10-953cbb082a14/test' /media/francesco/106bfc11-23d5-49c1-8c10-953cbb082a14/test ├── images │ ├── img1.jpg │ └── img2.jpg └── index.html
At first, on the remote server, we created this empty directory:
# mkdir /mnt/blockstorage/testBackup
However, let’s keep in mind that through the -exclude-wildcard option of gocryptfs, we excluded img2.jpg from the backup.
5.1. Initial Full Backup
During the first run of sync.sh, we must specify a robust encryption password, which must contain a nontrivial sequence of uppercase and lowercase characters, numbers, and special symbols, and be at least ten characters long:
$ ./sync.sh rsync --bwlimit setted to 0 kbytes/s (0 means no limit) The unencrypted directory /media/francesco/106bfc11-23d5-49c1-8c10-953cbb082a14/test contains local data to be backed up... Initializing read-only encrypted view of the unencrypted directory /media/francesco/106bfc11-23d5-49c1-8c10-953cbb082a14/test, without encrypting file names and symlink (to allow the recovery of individual files). Choose a password for protecting your files. Password: Repeat: Your master key is: [...] Password: Decrypting master key Filesystem mounted and ready. gocryptfs succeeded -> the decrypted dir /media/francesco/106bfc11-23d5-49c1-8c10-953cbb082a14/test is virtually encrypted in /home/francesco/toBeCopied sending incremental file list ./ gocryptfs.conf 378 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=3/5) index.html 47,344 100% 6.45MB/s 0:00:00 (xfr#2, to-chk=2/5) images/ images/img1.jpg 1,937,234 100% 16.50MB/s 0:00:00 (xfr#3, to-chk=0/5) rsync succeeded -> a full encrypted copy of /media/francesco/106bfc11-23d5-49c1-8c10-953cbb082a14/test is ready in [email protected][...]:/mnt/blockstorage/testBackup Press Enter to exit...
The output is verbose and self-explanatory. Everything went as expected.
5.2. Subsequent Backups
In subsequent runs of sync.sh, we must enter the encryption password specified in the first run. Since our script will send only new or updated files to the server, let’s try editing index.html before executing sync.sh again:
$ echo "abc1234" >> /media/francesco/106bfc11-23d5-49c1-8c10-953cbb082a14/test/index.html $ ./sync.sh [...] Password: Decrypting master key Filesystem mounted and ready. [...] sending incremental file list index.html 47,352 100% 6.36MB/s 0:00:00 (xfr#1, to-chk=2/5) [...]
This time, too, everything went as planned. Anyway, there’s a rare possibility that an error like the following may occur when rsync tries resuming a previously interrupted upload:
rsync error: error in rsync protocol data stream (code 12) at token.c(476) [sender=3.2.3]
Because of the infinite loop in which we put rsync, this error will continually repeat until we press CTRL+C. Its cause can be data corruption or rsync bugs. In that case, let’s delete the file causing the error to force rsync to upload it again from scratch.
5.3. Full Restore
Let’s run syncRestore.sh. The output will be verbose and self-explanatory:
$ ./syncRestore.sh rsync --bwlimit setted to 0 kbytes/s (0 means no limit) receiving incremental file list ./ gocryptfs.conf 378 100% 369.14kB/s 0:00:00 (xfr#1, to-chk=3/5) index.html 47,352 100% 126.34kB/s 0:00:00 (xfr#2, to-chk=2/5) images/ images/img1.jpg 1,937,234 100% 240.91kB/s 0:00:07 (xfr#3, to-chk=0/5) rsync succeeded -> a full copy of [email protected][...]:/mnt/blockstorage/testBackup encrypted data is ready in /home/francesco/toBeRestoreEncrypted Password: Decrypting master key Filesystem mounted and ready. gocryptfs succeeded -> the encrypted dir /home/francesco/toBeRestoreEncrypted is virtually decrypted in /home/francesco/toBeRestoreVirtualDecrypted You can now rsync the virtual dir /home/francesco/toBeRestoreVirtualDecrypted in a safe place When you done, remember to umount the virtual dir with: fusermount -u /home/francesco/toBeRestoreVirtualDecrypted Press Enter to exit...
Let’s do a quick check that the decrypted files are present:
$ tree /home/francesco/toBeRestoreVirtualDecrypted /home/francesco/toBeRestoreVirtualDecrypted ├── images │ └── img1.jpg └── index.html 1 directory, 2 files $ file /home/francesco/toBeRestoreVirtualDecrypted/index.html /home/francesco/toBeRestoreVirtualDecrypted/index.html: HTML document, Unicode text, UTF-8 text, with very long lines (1516) $ file /home/francesco/toBeRestoreVirtualDecrypted/images/img1.jpg /home/francesco/toBeRestoreVirtualDecrypted/images/img1.jpg: JPEG image data, JFIF standard 1.01, [...]
Everything went as planned. Let’s remember, however, that the decrypted files are within a virtual FUSE file system, so we must copy them elsewhere. Finally, let’s unmount this temporary file system:
$ fusermount -u /home/francesco/toBeRestoreVirtualDecrypted
Of course, we can also delete the /home/francesco/toBeRestoreEncrypted directory. It only makes sense to leave it if we want to run syncRestore.sh again.
5.4. Single File Restore
Let’s assume we only want to recover index.html without running syncRestore.sh. In that case, it’s enough to download that encrypted file, as well as gocryptfs.conf, and then decrypt it by manually running gocryptfs:
$ mkdir temp $ cd temp $ scp [email protected][...]:/mnt/blockstorage/testBackup/gocryptfs.conf ./ gocryptfs.conf 100% 378 1.0KB/s 00:00 $ scp [email protected][...]:/mnt/blockstorage/testBackup/index.html ./ index.html 100% 46KB 55.8KB/s 00:00 $ cd .. $ mkdir tempDecripted $ gocryptfs -ro temp tempDecripted Password: Decrypting master key Filesystem mounted and ready. $ file ./tempDecripted/index.html ./tempDecripted/index.html: HTML document, Unicode text, UTF-8 text, with very long lines (1516)
Again, we need to remember to copy the decrypted version of index.html where we wish before running fusermount -u ./tempDecripted.
6. A Small Trick if Internet Connectivity Is Unstable
Placing rsync inside an infinite loop assumes that any interruptions in Internet connectivity are temporary. Let’s suppose this is not so, as in the case of some VPNs that require manual intervention to reconnect. In that scenario, we can create a dedicated Bash script that uses the nmcli command to check our connection at regular intervals and restore it automatically.
Alternatively, if the loop is unnecessary or the interruption of connectivity always requires manual intervention, we can set the boolean variable $LOOP to false.
In this article, we saw how to create encrypted backups with rsync over ssh on a remote server. We could supplement our security with disk encryption solutions most Linux distributions offer.
Generally speaking, there is always the possibility of falling into data breaches or legal obligations hostile to our privacy. Even the most prominent and best-equipped companies in terms of security must consider this eventuality. For these reasons, not exposing our files in unencrypted form on third-party managed servers is now a must-have in both personal and business settings.