Continuously Sync Files One-Way on Linux

Last updated: March 18, 2024

Reviewed by: Ashley Frieze

1. Overview

In this tutorial, we’ll look at several methods to sync files one-way periodically or when we detect some changes in the files or directories.

We’ll be using a cron job to trigger the sync process with rsync/unison/FreeFileSync periodically and inotifywait or lsyncd to sync on change.

2. Sync Files Periodically

Let’s start with syncing files periodically with a cron job. First, we create a sync script. Then we set up a cron job to run the script every minute.

2.1. Creating a Sync Script

Let’s create a simple script and save it in sync-script.sh:

#!/bin/sh
date >> sync-script.log

The script writes the system date and time to sync-script.log file in append mode (notice the double greater-than sign >>, where the single > would overwrite the file). Later on, we’ll add some file synchronization to this script, but for now, let’s focus on getting it to work and scheduling it to run.

Next, we update our script file permissions to be executable, then run it:

$ sudo chmod 775 sync-script.sh
$ ./sync-script.sh

The script that we ran wrote the system date and time to sync-script.log:

$ cat sync-script.log
Thu 30 Jun 2022 02:14:45 PM

2.2. Setting up a Cron Job

Next, let’s create a cron job to run the script file every minute:

$ crontab -e
*/1 * * * * /bin/sh ~/sync-script.sh >> ~/sync-job.log 2>&1

We can check the script log to ensure the job is running as expected:

$ cat sync-script.log
Thu 30 Jun 2022 02:14:45 PM
Thu 30 Jun 2022 02:25:01 PM
Thu 30 Jun 2022 02:25:02 PM
Thu 30 Jun 2022 02:25:03 PM
...
Thu 30 Jun 2022 02:38:01 PM

We store any error message during script execution in the job log file:

$ cat sync-job.log

At this point, we now have a working cron job that runs the script file every minute.

3. Sync Periodically Using rsync

Now, let’s update our script to sync files periodically using rsync.

3.1. Updating the Script File

We’ll replace the date command with rsync:

#!/bin/sh
rsync -acu --delete ~/sample/source/ ~/sample/target/ >> ~/sync-script.log

Let’s review the rsync options that we used:

  • -a: enable archive mode; equivalent to –rlptgoD (no –H,-A,-X)
  • -c: skip object based on checksum, not on modified time and size
  • -u: skip newer files on the target
  • –delete: delete files from target not exist in source

3.2. Testing the Script File

We’ll now run the script to test it:

$ pwd
/home/baeldung
$ tree sample
sample
├── source
│   ├── file01.bin
│   ├── file02.bin
│   └── subdir1
│       └── file04.bin
└── target

3 directories, 3 files
$
$ ./sync-script.sh
$
$ tree sample
sample
├── source
│   ├── file01.bin
│   ├── file02.bin
│   └── subdir1
│       └── file04.bin
└── target
    ├── file01.bin
    ├── file02.bin
    └── subdir1
        └── file04.bin

4 directories, 6 files

Let’s delete a file (~/sample/source/file01.bin) from the source to see if it will also delete the same file from the target:

$ rm sample/source/file01.bin
$ tree sample
sample
├── source
│   ├── file02.bin
│   └── subdir1
│       └── file04.bin
└── target
    ├── file01.bin
    ├── file02.bin
    └── subdir1
        └── file04.bin

4 directories, 5 files
$ ./sync-script.sh
$ tree sample
sample
├── source
│   ├── file02.bin
│   └── subdir1
│       └── file04.bin
└── target
    ├── file02.bin
    └── subdir1
        └── file04.bin

4 directories, 4 files

The script removed the files in the target directory successfully. Our ~/sample/source directory will now sync to ~/sample/target directory every minute.

4. Sync Periodically Using Unison

Unison is another powerful file-synchronization tool. However, unlike rsync, besides having a one-way (mirror) mode, it also has a two-way synchronization mode.

Let’s update our sync script to use Unison command in mirror mode:

#!/bin/sh
unison -batch=true ~/sample/source/ ~/sample/target/ \
    -force ~/sample/source/ >> /home/baeldung/sync-script.log

The Unison command options that we use:

  • -batch=true: batch/non-interactive mode. Unison will use default settings without any interactive prompt
  • -force: sync in mirror mode

Our cron job will still run our script every minute. Only this time, the script will be using Unison command.

5. Sync Periodically Using FreeFileSync

FreeFileSync is a GUI-based file-synchronization tool. We need to generate the ffs_batch file, which contains our synchronization settings, in order to use it in batch/non-interactive mode for our script:

$ which FreeFileSync
/home/baeldung/.local/bin/FreeFileSync
$ vi sync-script.sh
#!/bin/sh
$ ./home/baeldung/.local/bin/FreeFileSync BatchRun.ffs_batch >> /home/baeldung/sync-script.log

We store our sync settings in the BatchRun.ffs_batch file and use it as a parameter to the FreeFileSync binary file.

6. Sync on Change Using inotifywait

Instead of using a cron job to run the sync command periodically, which may cause memory or processor overhead, let’s now look at using inotifywait to do file sync only when we make some changes to the file or directory.

6.1. Installing inotifywait

inotifywait is part of the inotify-tools package, which is available on many Linux core repositories. For example, on Debian/Ubuntu, we can run:

$ apt install inotify-tools

On Fedora:

$ dnf -y install inotify-tools

On Arch and CentOS/RHEL:

$ yum install -y epel-release && yum update
$ yum install inotify-tools

The inotify repository has a complete list of Linux distros that have the inotify-tools package.

6.2. Using inotifywait

Let’s create a script to use inotifywait to run the sync command only when there’s a change to the file or directory:

#!/bin/bash

if [ -z "$(which inotifywait)" ]; then
    echo "inotifywait not installed."
    echo "In most distros, it is available in the inotify-tools package."
    exit 1
fi

counter=0;

function execute() {
    counter=$((counter+1))
    echo "Detected change n. $counter"
    eval "$@"
}

inotifywait --recursive --monitor --format "%e %w%f" \
--event modify,move,create,delete /home/baeldung/sample/source \
| while read changed; do
    echo $changed
    execute "$@"
done

The script above waits for filesystem events (CREATE, MODIFY, MOVE, DELETE) and acts accordingly by running the execute function.

Let’s update the execute function to run the rsync/Unison/FreeFileSync command from the previous section to sync our directories:

function execute() {
    counter=$((counter+1))
    echo "Detected change n. $counter"
    eval "$@"
    if test $counter -gt 100
    then
        rsync -cau /home/baeldung/sample/source/ /home/baeldung/sample/target/ \
            >> /home/baeldung/sync-script.log 2>&1
    fi
}

Since we don’t want to run the sync command every time we make some changes, we configured the script to run the sync command after there have been over 100 events.

6.3. Troubleshooting a Lack of Events

Our script should work just fine, but what will happen if we rarely change the file or directory and inotifywait receives less than 100 events? Our directories won’t be in sync until we make enough changes.

To handle this case, we have a few options:

  • update our script, so we only wait for fewer events
  • add a timer to sync the directories every few minutes
  • combine the script with a cron job that runs every 30 minutes or hourly

7. Sync on Change Using lsyncd

lsyncd is a daemon that uses the filesystem event interface (inotify or fsevents) to detect changes to local files or directories. It uses rsync as the default synchronization method.

7.1. Installing lsyncd

lsyncd is available on many Linux core repositories. For example, on Debian or Ubuntu, we can run:

$ apt install lsyncd

On Fedora:

$ dnf -y install lsyncd

On CentOS/RHEL 7:

$ yum install lsyncd

We can also download lsyncd from their website directly. We should note that lsyncd 2.2.1 or later requires rsync >= 3.1 on all source and target machines.

7.2. Using lsyncd

Let’s set up a local lsync:

$ lsyncd -rsync /home/baeldung/sample/source /home/baeldung/sample/target

If the target directory is on a different machine:

$ lsyncd -rsyncssh /home/baeldung/sample/source/ remotehost.org target-path/

The command above will copy/mirror the source directory recursively to the target directory:

$ tree sample
sample
└── source
    ├── file01.bin
    ├── file02.bin
    └── subdir1
        └── file03.bin

2 directories, 3 files
$ lsyncd -rsync /home/baeldung/sample/source /home/baeldung/sample/target
15:41:03 Normal: --- Startup, daemonizing ---
$ tree sample
sample
├── source
│   ├── file01.bin
│   ├── file02.bin
│   └── subdir1
│       └── file03.bin
└── target
    ├── file01.bin
    ├── file02.bin
    └── subdir1
        └── file03.bin

4 directories, 6 files

If we edit files in the source directory, lsyncd will automatically reflect it in the target directory:

$ cat sample/source/file01.bin
This is line 1
$ cat sample/target/file01.bin
This is line 1
$ echo "This is line 2" >> sample/source/file01.bin
$ cat sample/source/file01.bin
This is line 1
This is line 2
$ cat sample/target/file01.bin
This is line 1
...
$ cat sample/target/file01.bin
This is line 1
$ cat sample/target/file01.bin
This is line 1
This is line 2

lsyncd aggregates events up to 1000 separate events, or a 15-second delay before synchronizing, whichever happens first, so our changes may not be synced immediately.

7.3. lsyncd Options

Let’s try configuring lsyncd by creating its config file lsyncd-config.cfg:

settings{
    logfile = "/home/baeldung/lsyncd-log.log",
    statusFile = "/home/baeldung/lsyncd-status.stat",
}
sync{
    default.rsync,
    source="/home/baeldung/sample/source",
    target="/home/baeldung/sample/target",
}

We pass the config file as a parameter to the lsyncd command:

$ lsyncd lsyncd_config.cfg
15:56:54 Normal: --- Startup, daemonizing ---
$ ls -l lsyncd*
-rw-r--r-- 1 baeldung baeldung 275 Jul 2 15:55 lsyncd_config.cfg
-rw-r--r-- 1 baeldung baeldung 356 Jul 2 15:56 lsyncd-log.log
-rw-r--r-- 1 baeldung baeldung 287 Jul 2 15:56 lsyncd-status.stat

Let’s review the options that we used in the config file:

  • settings: global config options which will be used by all syncs
  • logfile: log file path
  • statusFile: status file path
  • sync: config options for each sync process
  • default.rsync: file synchronization tool
  • source: source directory
  • target: target directory

7.4. Sync From One Source to Multiple Targets

lsyncd has the feature of syncing from one source to multiple targets. Let’s update our config file:

settings{
    logfile = "/home/baeldung/lsyncd-log.log",
    statusFile = "/home/baeldung/lsyncd-status.stat",
}
sync{
    default.rsync,
    source="/home/baeldung/sample/source",
    target="/home/baeldung/sample/target",
}
sync{
    default.rsync,
    source="/home/baeldung/sample/source",
    target="/home/baeldung/sample/target2",
}
sync{
    default.rsync,
    source="/home/baeldung/sample/source",
    target="/home/baeldung/sample/target3",
}

Then we  run lsyncd with the new config file:

$ tree sample
sample
└── source
    ├── file01.bin
    ├── file02.bin
    └── subdir1
        └── file03.bin

2 directories, 3 files
$ lsyncd lsyncd-config.cfg
16:08:58 Normal: --- Startup, daemonizing ---
$ tree sample
sample
├── source
│   ├── file01.bin
│   ├── file02.bin
│   └── subdir1
│       └── file03.bin
├── target
│   ├── file01.bin
│   ├── file02.bin
│   └── subdir1
│       └── file03.bin
├── target2
│   ├── file01.bin
│   ├── file02.bin
│   └── subdir1
│       └── file03.bin
└── target3
    ├── file01.bin
    ├── file02.bin
    └── subdir1
        └── file03.bin

8 directories, 12 files

The lsyncd website has a complete list of config options with detailed explanations and examples.

8. Conclusion

In this article, we saw how to continuously sync files one-way periodically or on change.

We first utilized a cron job to periodically run file sync commands, such as rsync/Unison/FreeFileSync.

Then we used inotifywait to detect filesystem events as a trigger to run the file sync command.

Lastly, we used the lsyncd tool, which uses the filesystem event interface (inotify or fsevents) to detect changes, as well as a periodic sync, which provided the best combination of all techniques.

Comments are open for 30 days after publishing a post. For any issues past this date, use the Contact form on the site.