Synchronize Linux Directories in Both Directions

1. Overview

Often, we need to back up files or directories to another place on a local machine, or between local and remote machines. We can do it manually by copy-and-paste, which is time-consuming and error-prone, or we can utilize some command or program to do it.

In this tutorial, we’ll test several commands or programs to synchronize directories in both directions on a single machine and compare the results.

We tested the code using 64-bit rsync 3.1.3, 64-bit Unison 2.51.4 with OCaml 4.05.0 binary for Debian, Zaloha2, and FreeFileSync 11.16 binary for Linux running on 64-bit Debian 10.10 (Buster) with GNU bash 5.0.3.

2. Using the rsync Command

2.1. Installation

rsync is commonly found on Unix-like operating systems. It’s an open-source project under GNU GPL, and its source code or binary can be downloaded from its homepage.

2.2. Usage

rsync works in one direction, so we need to run it twice to sync directories in both directions.

Let’s give it a go. First, let’s view the structure of source and target directories:

$ tree
.
├── source
│   ├── file01.bin
│   ├── file02.bin
│   └── file03.bin
└── target
    ├── file0A.bin
    ├── file0B.bin
    └── file0C.bin

2 directories, 6 files

Now, let’s sync from source to target:

$ rsync -Pcauv source/ target/
sending incremental file list
./
file01.bin
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=2/4)
file02.bin
              0 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=1/4)
file03.bin
              0 100%    0.00kB/s    0:00:00 (xfr#3, to-chk=0/4)

sent 297 bytes  received 76 bytes  746.00 bytes/sec
total size is 0  speedup is 0.00

And check our intermediate results:

$ tree
.
├── source
│   ├── file01.bin
│   ├── file02.bin
│   └── file03.bin
└── target
    ├── file01.bin
    ├── file02.bin
    ├── file03.bin
    ├── file0A.bin
    ├── file0B.bin
    └── file0C.bin

2 directories, 9 files

Then, we sync from target to source:

$ rsync -Pcauv target/ source/
sending incremental file list
file0A.bin
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=2/7)
file0B.bin
              0 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=1/7)
file0C.bin
              0 100%    0.00kB/s    0:00:00 (xfr#3, to-chk=0/7)

sent 401 bytes  received 73 bytes  948.00 bytes/sec
total size is 0  speedup is 0.00

Now, let’s check our final result:

$ tree
.
├── source
│   ├── file01.bin
│   ├── file02.bin
│   ├── file03.bin
│   ├── file0A.bin
│   ├── file0B.bin
│   └── file0C.bin
└── target
    ├── file01.bin
    ├── file02.bin
    ├── file03.bin
    ├── file0A.bin
    ├── file0B.bin
    └── file0C.bin

2 directories, 12 files

Let’s review the options that we use:

-P: display statistics of the whole transfer rather than individual files
-c: skip object based on checksum, not on modified time and size
-a: enable archive mode, equals to:
- -r: recurse into directories
- -l: copy symlinks as symlinks
- -p: preserve permissions
- -t: preserve modification times
- -g: preserve group
- -o: preserve owner
- -D: preserve device files and special files
-u: skip newer files on the target
-v: display detailed information of what is being done

We can also do a dry run using the -n option, which will simulate the sync with no changes made, in order to prevent data loss:

$ rsync -Pcauvn source/ target/
sending incremental file list
./
file01.bin
file02.bin
file03.bin

sent 189 bytes  received 28 bytes  434.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

3. Using Unison Program

3.1. Installation

Unison source code or binary download information is available on its homepage, where it points to this GitHub repository at the time of writing.

We’re using 64-bit Unison 2.51.4 with OCaml 4.05.0 binary, which gives us this package:

$ tree
.
├── bin
│   ├── unison
│   ├── unison-fsmonitor
│   └── unison-gtk2
├── LICENSE
└── README.md

1 directory, 5 files

3.2. Usage

We can use the GUI version of Unison by running the unison-gtk executable file. For this article, we’ll use text-based Unison, which is the unison executable file.

Let’s view the directory structure of our source and target directories:

$ tree
.
├── source
│   ├── file01.bin
│   ├── file02.bin
│   └── file03.bin
└── target
    ├── file0A.bin
    ├── file0B.bin
    └── file0C.bin

2 directories, 6 files

When we first do the sync, we’ll get this interactive prompt:

$ ./unison ~/source/ ~/target/
Unison 2.51.4 (ocaml 4.05.0): Contacting server...
Looking for changes
Warning: No archive files were found for these roots, whose canonical names are:
	~/source
	~/target
This can happen either because this is the first time you have synchronized these roots, or because you have upgraded Unison to a new version with a different archive format.  
...
Donations to the Unison project are gratefully accepted: http://www.cis.upenn.edu/~bcpierce/unison

Press return to continue.[<spc>]

We can press Enter to continue, or q to quit. If we press Enter, it will ask for confirmation for each change that we want to do.

We press f or Enter to confirm each change. We can press > or < to force the change to be propagated from left to right, or right to left, or press / to skip the file and leave both objects alone.

When we reach the end of the list, Unison will ask one more time if we want to proceed with the changes, and we can press y to proceed, or q to terminate.

Every time Unison asks for input, we can press? to list all possible responses and their meanings:

Reconciling changes

source         target             
file     ---->            file01.bin  [f] f
file     ---->            file02.bin  [f] f
file     ---->            file03.bin  [f] f
         <---- file       file0A.bin  [f] f
         <---- file       file0B.bin  [f] f
         <---- file       file0C.bin  [f] f

Proceed with propagating updates? [] y
Propagating updates


Unison 2.51.4 (ocaml 4.05.0) started propagating changes at 15:29:09.85 on 06 Jan 2022
[BGN] Copying file01.bin from ~/source to ~/target
[END] Copying file01.bin
[BGN] Copying file02.bin from ~/source to ~/target
[END] Copying file02.bin
[BGN] Copying file03.bin from ~/source to ~/target
[END] Copying file03.bin
[BGN] Copying file0A.bin from ~/target to ~/source
[END] Copying file0A.bin
[BGN] Copying file0B.bin from ~/target to ~/source
[END] Copying file0B.bin
[BGN] Copying file0C.bin from ~/target to ~/source
[END] Copying file0C.bin
Unison 2.51.4 (ocaml 4.05.0) finished propagating changes at 15:29:09.86 on 06 Jan 2022, 0.012 s


Saving synchronizer state
Synchronization complete at 15:29:09  (6 items transferred, 0 skipped, 0 failed)

Once it’s done, if we try to sync again, it will say nothing to sync:

$ ./unison ~/source/ ~/target/
Unison 2.51.4 (ocaml 4.05.0): Contacting server...
Looking for changes
Reconciling changes
Nothing to do: replicas have not changed since last sync.

If we delete or modify some files and try to sync again, we need to do the confirmation again:

$ rm source/file0B.bin 
$ echo "edit file" > source/file0C.bin 
$ tree
.
├── source
│   ├── file01.bin
│   ├── file02.bin
│   ├── file03.bin
│   ├── file0A.bin
│   └── file0C.bin
└── target
    ├── file01.bin
    ├── file02.bin
    ├── file03.bin
    ├── file0A.bin
    ├── file0B.bin
    └── file0C.bin

2 directories, 11 files

$ ./unison ~/source/ ~/target/
Unison 2.51.4 (ocaml 4.05.0): Contacting server...
Looking for changes
Reconciling changes

source         target             
deleted  ---->            file0B.bin  [f] 
changed  ---->            file0C.bin  [f] 

Proceed with propagating updates? [] y
Propagating updates


Unison 2.51.4 (ocaml 4.05.0) started propagating changes at 15:50:16.93 on 06 Jan 2022
[BGN] Updating file file0C.bin from ~/source to ~/target
[END] Updating file file0C.bin
[BGN] Deleting file0B.bin from ~/target
[END] Deleting file0B.bin
Unison 2.51.4 (ocaml 4.05.0) finished propagating changes at 15:50:16.93 on 06 Jan 2022, 0.003 s


Saving synchronizer state

If we want to immediately proceed to sync with default Unison settings without any interactive prompt, where non-conflicting changes will be propagated and conflicts will be skipped, we can pass -batch=true as an option:

$ ./unison -batch=true ~/source/ ~/target/
Unison 2.51.4 (ocaml 4.05.0): Contacting server...
Looking for changes
Reconciling changes
changed  ---->            file0C.bin  
source       : changed file       modified on 2022-01-06 at 16:40:13  size 11        rw-r--r--
target       : unchanged file     modified on 2022-01-06 at 16:28:13  size 11        rw-r--r--
Propagating updates
Unison 2.51.4 (ocaml 4.05.0) started propagating changes at 16:40:35.93 on 06 Jan 2022
[BGN] Updating file file0C.bin from ~/source to ~/target
[END] Updating file file0C.bin
Unison 2.51.4 (ocaml 4.05.0) finished propagating changes at 16:40:35.93 on 06 Jan 2022, 0.000 s
Saving synchronizer state
Synchronization complete at 16:40:35  (1 item transferred, 0 skipped, 0 failed)

4. Using Zaloha2 Bash Script

4.1. Installation

Zaloha2 is a small and simple directory synchronizer in a form of a Bash script. We can download it from its GitHub repository.

Once we have downloaded the project, we will get the following package:

$ tree
.
├── DOCUMENTATION.md
├── LICENSE
├── README.md
├── Simple_Demo_screenshot.png
├── Simple_Demo_step1.sh
├── Simple_Demo_step2.sh
├── Simple_Demo_step3.sh
├── Simple_Demo_step4.sh
├── Simple_Demo_step5.sh
├── Simple_Demo_step6.sh
├── Simple_Demo_step7.sh
└── Zaloha2.sh

0 directories, 12 files

We need to make Zaloha2.sh executable, and then we’re ready to use it:

$ sudo chmod u+x Zaloha2.sh

4.2. Usage

Using Zaloha2 is pretty straightforward. First, let’s say we have the following directory structure:

$ tree
.
├── source
│   ├── file01.bin
│   ├── file02.bin
│   └── file03.bin
└── target
    ├── file0A.bin
    ├── file0B.bin
    └── file0C.bin

2 directories, 6 files

We then run the sync process:

$ ./Zaloha2.sh --sourceDir="/home/baeldung/source" --backupDir="/home/baeldung/target" --color --revUp --revNew --noRemove --sha256 --findSourceOps='( -type d -a -name subdir ) -prune -o' --findSourceOps='( -type d -a -name .svn ) -prune -o' --findSourceOps='( -type d -a -name .cache ) -prune -o' --findGeneralOps=

TO BE COPIED TO /home/baeldung/target/
===========================================
NEW       file01.bin
NEW       file02.bin
NEW       file03.bin
MKDIR     subdir2
NEW       subdir2/file05.bin

Execute above listed copies to /home/baeldung/target/ ?
[Y/y=Yes, S/s=do nothing and show further, other=do nothing and abort]: y

    mkdir /home/baeldung/target/subdir2
    cp --preserve=timestamps /home/baeldung/source/file01.bin /home/baeldung/target/file01.bin
    cp --preserve=timestamps /home/baeldung/source/file02.bin /home/baeldung/target/file02.bin
    cp --preserve=timestamps /home/baeldung/source/file03.bin /home/baeldung/target/file03.bin
    cp --preserve=timestamps /home/baeldung/source/subdir2/file05.bin /home/baeldung/target/subdir2/file05.bin

TO BE REVERSE-COPIED TO /home/baeldung/source/
===========================================

FROM COMPARING CONTENTS OF FILES: TO BE COPIED TO /home/baeldung/target/
===========================================
$
$ tree
.
├── source
│   ├── file01.bin
│   ├── file02.bin
│   ├── file03.bin
│   ├── subdir
│   │   └── file04.bin
│   └── subdir2
│       └── file05.bin
└── target
    ├── file01.bin
    ├── file02.bin
    ├── file03.bin
    ├── file0A.bin
    ├── file0B.bin
    ├── file0C.bin
    └── subdir2
        └── file05.bin

5 directories, 12 files

Let’s examine the options that we use:

–sourceDir: source directory
–backupDir: target directory
–color: to colorize the changes during interactive prompt
–revUp: reverse-update; if there are newer objects in target, copy them to the source
–revNew: reverse-new; if there are objects in the target that don’t exist in the source and they are newer than the last run of Zaloha2, copy them to the source
–noRemove: do not remove objects in target; only add or update existing objects in the target
–sha256: besides comparing file size and modified date, check file SHA256 checksum
–noProgress: do not display messages from the analysis phase
–findSourceOps: exclude certain subdirectories from sync. If such subdirectory exists in the target, it will be deleted
–findGeneralOps: if it is empty, it will disable internally defined defaults that exclude certain directories (“trash” or “lost+found”). It can also be used instead of findSourceOps to generally exclude files or directory

Unlike rsync or Unison, Zaloha2 doesn’t reverse copy files or directories from target to source initially, unless those files or directories are newer than the last run of Zaloha2.

So, let’s update some files in the target directory, then run Zaloha2 again to trigger reverse copy:

$ echo "edit" > target/file0B.bin 
$ echo "edit" > target/file0C.bin 
$ echo "edit" > target/subdirB/file0E.bin 
$
$ ./Zaloha2.sh --sourceDir="/home/baeldung/source" --backupDir="/home/baeldung/target" --color --revUp --revNew --noRemove --sha256 --findSourceOps='( -type d -a -name subdir1 ) -prune -o' --findSourceOps='( -type d -a -name subdirA ) -prune -o' --findGeneralOps=

TO BE COPIED TO /home/baeldung/target/
===========================================

TO BE REVERSE-COPIED TO /home/baeldung/source/
===========================================
REV.NEW   file0A.bin
REV.NEW   file0B.bin
REV.NEW   file0C.bin
REV.MKDI  subdirB
REV.NEW   subdirB/file0E.bin

Execute above listed reverse-copies to /home/baeldung/source/ ?
[Y/y=Yes, S/s=do nothing and show further, other=do nothing and abort]: y

    '[' '!' -e /home/baeldung/source/file0A.bin ']'
    '[' '!' -e /home/baeldung/source/file0B.bin ']'
    '[' '!' -e /home/baeldung/source/file0C.bin ']'
    '[' '!' -e /home/baeldung/source/subdirB ']'
    mkdir /home/baeldung/source/subdirB
    '[' '!' -e /home/baeldung/source/subdirB/file0E.bin ']'
    cp --preserve=timestamps /home/baeldung/target/file0A.bin /home/baeldung/source/file0A.bin
    cp --preserve=timestamps /home/baeldung/target/file0B.bin /home/baeldung/source/file0B.bin
    cp --preserve=timestamps /home/baeldung/target/file0C.bin /home/baeldung/source/file0C.bin
    cp --preserve=timestamps /home/baeldung/target/subdirB/file0E.bin /home/baeldung/source/subdirB/file0E.bin

FROM COMPARING CONTENTS OF FILES: TO BE COPIED TO /home/baeldung/target/
===========================================
$
$ tree
.
├── source
│   ├── file01.bin
│   ├── file02.bin
│   ├── file03.bin
│   ├── file0A.bin
│   ├── file0B.bin
│   ├── file0C.bin
│   ├── subdir1
│   │   └── file04.bin
│   ├── subdir2
│   │   └── file05.bin
│   └── subdirB
│       └── file0E.bin
└── target
    ├── file01.bin
    ├── file02.bin
    ├── file03.bin
    ├── file0A.bin
    ├── file0B.bin
    ├── file0C.bin
    ├── subdir2
    │   └── file05.bin
    ├── subdirA
    │   └── file0D.bin
    └── subdirB
        └── file0E.bin

8 directories, 18 files

If we want to set up automatic/non-interactive mode, there are steps that are a bit complicated, albeit flexible, to implement, which the author suggests in the documentation.

What we need to do is to run the command with an extra option –noExec, which will only generate the scripts (the scripts are stored in the .Zalora_metadata in the target directory). Then, we need to write a wrapper script to execute those scripts. Please refer to Zaloha2 documentation in the Automatic Operations section.

Zaloha2 project is still in active development — use with caution.

5. Using FreeFileSync Program

5.1. Installation

FreeFileSync is open-source, multi-platform software. It’s GUI-based but also provides a feature to do batch/non-interactive mode.

We can download the FreeFileSync installer script from its homepage and run it, which will create the Desktop shortcut.

5.2. Usage

This is the main UI of FreeFileSync:

The interface is quite intuitive:

Change comparison settings, such as compare based on the file modified time and size, or file content, or file size.
Start comparison between the two directories.
Filter: we can include or exclude certain files or directories based on the file name, extension, size, or modified time.
Change synchronization settings: we can synchronize in two directions (Two way), one-way (Mirror), update only (Update), or custom.
Start synchronization.
Source directory.
Target directory.

If we want to create a batch/non-interactive mode, first we need to select source and target directories, then click on File > Save as batch job…, which will show the following dialog:

We need to disable the Progress dialog option and enable the Ignore errors option. Otherwise, it will display the interactive/error message box and will wait for interaction indefinitely.

After we create the batch job file, e.g. BatchRun.ffs_batch, we need to set FreeFileSync to run it.

For example, we can set the cron job to run it every day at 1AM:

$ crontab -e

0 1 * * * <FreeFileSync installation path>/FreeFileSync <batch file path>/BatchRun.ffs_batch

6. Comparison

6.1. Features

Tool	Programming Language	License	Bidirectional	Delta Copying	Detect Conflict	Propagate Deletes	Revision Control	Scheduling or Service
rsync	C	GPL v3	Yes	Yes	Yes	Yes using –delete	Yes using –backup and a time-stamped –suffix	Yes via OS
Unison	Mainly OCaml	GPL	Yes	Yes	Yes	Yes	Yes	Yes via OS
Zaloha2	Bash	MIT	Yes	Yes	No	Yes	Yes	Yes via OS
FreeFileSync	C++	GPL	Yes	Yes	Yes	Yes	Yes	Yes via OS

6.2. Benchmark

We measured the performance on the following system and data:

Dell Latitude Intel Core i7-6600U CPU @ 2.60GHz × 4, 16GB RAM
Source directory: Seagate external HDD 5TB, 7200rpm, SATA III, ext4
Target directory: Seagate external HDD 1.5TB, 7200rpm, SATA III, NTFS
Data synchronized: 10GB, 11,780 files
OS: Debian Buster

Here’s the number of files and total size of the source directory:

$ find source -type f | wc -l
11780
$
$ du source -hc --max-depth=0
1.3G source/subdir1
3.3G source/subdir2
5.5G source/subdir3
10G source
10G total

We tested each approach by running the commands three times (except for rsync – six times – as it is not bidirectional):

For the first run, we synced/mirrored data from source to target directory.
For the second run, there’s nothing to do, so all tools returned immediately.
Then for the third run, first we deleted some data (delete subdir1) in source and added some data (rename subdir1 to subdir4) in target, then we synced the directories.

The options that we used for each command:

rsync

$ rsync -PScauv --info=progress2 source/ /media/baeldung/target/
$ rsync -PScauv --info=progress2 target/ /media/baeldung/source/

Unison

$ ./unison -batch=true /media/baeldung/source/ /media/baeldung/target/

Zaloha2

$ ./Zaloha2.sh --sourceDir="/media/baeldung/source" --backupDir="/media/baeldung/target" --color --revUp --revNew --noRemove --noRestore --mawk --findGeneralOps=

FreeFileSync

$ which FreeFileSync
/home/baeldung/.local/bin/FreeFileSync
$ ./home/baeldung/.local/bin/FreeFileSync BatchRun.ffs_batch

Results:

Tool	First Run	Second Run	Third Run
rsync	15:44	00:01	15:48
Unison	07:31	00:01	00:04
Zaloha2	05:22	00:01	Failed (no reverse update)
FreeFileSync	05:10	00:01	00:13

7. Conclusion

Of all four tools that we tested, we found that Unison is the best for synchronizing directories in both directions.

Unison may take longer when we run it for the first time, but the subsequent run was the fastest compared to the other tools in terms of performance. It supports conflict detection, is resilient to failure (it has revision control, useful for rollback), is bidirectional, and has a command-line as well as a GUI interface.

Coming in a close second is FreeFileSync, which is also great with its intuitive GUI.

Full Archive

About Baeldung