rsync and Hidden Files

1. Overview

In this tutorial, we discuss what we have to keep in mind when moving data from one directory to another, for which one helpful tool is the rsync command.

A typical use case, for example, is when we develop a website staging from development to live system.

2. Problem With rsync and dotfiles

Let’s assume a typical scenario. We have two folders with web content. One is the development path. The other one is the production path:

$ ls -las
total 16
4 drwxrwxr-x  4 tracer tracer 4096 Feb 28 13:05 .
4 drwxr-xr-x 39 tracer tracer 4096 Feb 28 13:11 ..
4 drwxrwxr-x  2 tracer tracer 4096 Feb 28 12:57 dev
4 drwxrwxr-x  2 tracer tracer 4096 Feb 28 13:05 prod

The content of our dev environment:

$ ls -las dev
total 8
4 drwxrwxr-x 2 tracer tracer 4096 Feb 28 12:57 .
4 drwxrwxr-x 4 tracer tracer 4096 Feb 28 13:05 ..
0 -rw-rw-r-- 1 tracer tracer    0 Feb 28 12:57 .htaccess
0 -rw-rw-r-- 1 tracer tracer    0 Feb 28 12:57 index.html

When we try to use the rsync command in this way, it will give us an unexpected result:

$ rsync -av dev/* prod
sending incremental file list
index.html

As we can see, rsync omits the .htaccess file, or at least, it seems to be the case.

3. Solution

When we leave out the asterisk and run the command again:

rsync -av dev/ prod
sending incremental file list
./
.htaccess
index.html

We can see that even the hidden file .htacces is copied.

4. Explanation

In this case, our problem is related to the way the shell works. There is a mechanism called globbing, which expands asterisks to filenames before the rsync program is actually called.

Let’s assume we have a few more files:

ls -las dev
total 8
4 drwxrwxr-x 2 tracer tracer 4096 Feb 28 13:22 .
4 drwxrwxr-x 4 tracer tracer 4096 Feb 28 13:05 ..
0 -rw-rw-r-- 1 tracer tracer    0 Feb 28 12:57 .htaccess
0 -rw-rw-r-- 1 tracer tracer    0 Feb 28 13:22 about.html
0 -rw-rw-r-- 1 tracer tracer    0 Feb 28 13:22 home.html
0 -rw-rw-r-- 1 tracer tracer    0 Feb 28 12:57 index.html

Let’s issue the command:

rsync -av dev/* prod

The shell expands the content from the directory before sending it to the rsync command. In reality, the call looks like this:

rsync -av about.html home.html index.html prod

So, the globbing of the shell ignores dotfiles, which are also called hidden files. If we just add the directory itself as a parameter, no globbing takes place, and rsync even copies dotfiles.

5. Conclusion

In this article, we looked at how to include dotfiles when syncing folders with the rsync command.