1. Overview

Sometimes we want to transfer files from one Linux machine to another securely. Perhaps we want to upload some files to a production server or take a backup.

In this article, we’ll look at different tools for transferring files between Linux machines over ssh, the most popular protocol for remote connection between Linux machines.

We’ll look at the two most popular file transfer tools: scp and rsync.

2. Basic Tool Usage

2.1. A Quick Example

Let’s imagine we want to copy the file text.log from our local system to the path /var/log on a remote Linux server at IP address 111.111.111.111, and we want to use root user credentials.

This can be achieved with scp or rsync using the same basic syntax:

scp text.log [email protected]:/var/log
rsync text.log [email protected]:/var/log

Let’s understand what these commands have in common before we learn their variations and differences.

2.2. The General Syntax

The general syntax for scp or rsync is:

scp/rsync [OPTION] [user@][SRC_HOST:]Source_File [user@][DEST_HOST:]Destination_file

The OPTION section allows us to use command line switches to modify the behavior of the copy operation. The two paths for source and destination allow us to specify the path to the file at both sides of the copy operation, as well as the user account(s) to use.

In the above example, the local file is copied to a remote location — [email protected]. As we’ll see later, we can use the remote format to describe source or target, allowing these commands to both upload files to a server and download files from a server.

2.3. Important Things to Note

There are few points to remember when we’re transferring files over ssh using any tool:

  • The scp/rsync command will ask for the user password to login to the remote server if we don’t have auto-login setup by exchanging RSA keys
  • We don’t need to provide our username or server IP for our local server
  • We can either provide an absolute path or relative path to the source/destination file, where relative paths are relative to the current working directory or the remote user’s home directory

2.4. Specifying a Filename

It’s not compulsory to provide a filename at the destination path. If we don’t provide a filename, the source file’s name will be used.

In our first example, the file will be created in the target directory with the name text.log. If we’d wanted the file to be stored with a different name, we could’ve provided a filename as part of the destination path.

Using the implicit filename, we can copy a file from remote Linux server into the current directory with its original name:

scp [email protected]:/var/log/text.log .
rsync [email protected]:/var/log/text.log .

Here, the “.” parameter represents the current working directory. We can also provide a different path if want:

scp [email protected]:/var/log/text.log /opt/path/mytextlog.log
rsync [email protected]:/var/log/text.log /opt/path/mytextlog.log

2.5. Copying Between Servers

We can even use these tools to copy between two remote servers. For example, let’s copy the text.log file from server 111.111.111.111. to server 222.222.222.222 for the root user:

scp [email protected]:/var/log/text.log [email protected]:/var/log
rsync [email protected]:/var/log/text.log [email protected]:/var/log

Now that we have a basic understanding of scp and rsync, let’s look at them in more depth.

3. scp (Secure Copy)

scp stands for Secure Copy and is used to transfer files over an ssh connection. It is a raw copy, meaning it will just read the data from the source folder and write it in the destination folder. So, if we are looking for a quick copy that shouldn’t take more than a few minutes, then we should go for scp.

If we’re transferring a large file, we should make sure that we have a stable internet connection. As scp is not resumable, time can be lost if the connection drops during a large transfer, and the entire copy operation will have to be restarted.

3.1. Installation

Most Linux distributions come with scp pre-installed. We can easily check if scp is installed or not:

which scp

We’ll see the path to the scp command if it’s already installed. An empty response means that scp is not installed.

If we don’t already have it, then to install scp, we need to install the OpenSSH client package from the official package repository:

sudo apt-get install openssh-clients    # For Debian based systems like Ubuntu 
sudo yum install openssh-clients        # For Red Hat based systems like CentOS and Fedora

Let’s go through a few common scenarios for using scp.

3.2. Transfer Multiple Files

We can provide a list of files separated by space to transfer multiple files:

scp text.log text1.log text2.log [email protected]:/var/log

We can also use the -r option to copy the whole directory recursively. For example, to copy all files from the /home/app/log/ directory:

scp -r /home/app/log [email protected]:/var/log

We can use * to send files matching a particular naming pattern. For example, we have four files text.log, text2.log, app.log, and app2.log, and we want to send only files starting with app:

scp /home/app/log/app* [email protected]:/var/log

3.3. Transfer Files Using Identity Files

Key-based authentication is preferred in most Linux distributions. In the scp command, we can specify the identity file using the -i option:

scp -i key.pem text.log [email protected]:/var/log

Many system administrators use key-based authentication to transfer files since it doesn’t require the user credentials on every transfer.

3.4. On-the-Fly Compression

We can reduce transfer time by using on-the-fly compression with the -C option. This will compress and decompress the file at source and destination paths respectively:

scp -C text.log [email protected]:/var/log

3.5. Limit Bandwidth During Transfer

We can specify bandwidth in Kbit/s with the -l option:

scp -l 500 text.log [email protected]:/var/log

3.6. Verbose Mode for Debugging

We can enable verbose mode with the -v option:

scp -v text.log [email protected]:/var/log

The output would look something like:

Executing: program /usr/bin/ssh host prod-reporting-b, user jain, command scp -v -t /tmp 
OpenSSH_7.9p1, LibreSSL 2.7.3 
debug1: Reading configuration data /etc/ssh/ssh_config 
debug1: /etc/ssh/ssh_config line 48: Applying options for * 
debug1: Connecting to prod-reporting-b port 22. 
debug1: Connection established. 
------ 
Transferred: sent 2856, received 2932 bytes, in 2.1 seconds 
Bytes per second: sent 1366.8, received 1403.1 
debug1: Exit status 0

Verbose (debug) output prints the exact steps scp is executing, which is useful for debugging connections, authentication, and configuration problems.

4. rsync (Remote Synchronization)

rsync is fast and versatile. It’s good for synchronizing and smartly transferring files because it keeps track of how much data has been copied and how much is left.

We should use rsync when we have big files to transfer — even if there are network connection problems, data copied up to that point won’t be lost, and when the connection is restored, the copy will resume from that point.

rsync is faster than the scp command when we have to transfer the same file multiple times as it uses a remote-update protocol, which allows transferring just the differences between two sets of files.

4.1. Installation

Let’s check if rsync is installed by invoking it to show its version:

rsync --version

If we get an error, then we can install it:

sudo apt-get install rsync    # For Debian based systems like Ubuntu 
sudo yum install rsync        # For Red Hat based systems like CentOS and Fedora

Let’s go through some common scenarios where we can use rsync.

4.2. Do a Dry Run

Since rsync is so powerful, it can be dangerous to use it without checking what it’s about to do. Fortunately, rsync provides the –dry-run option, which performs a trial run without actually transferring files:

rsync --dry-run text.log [email protected]:/var/log

4.3. Copy Attributes and Metadata

rsync provides the -a option, which copies symbolic links, modification times, group, ownership, and permissions along with the file content.

Another important option is –h, which will print output in a human-readable form. This can be especially helpful when running along with the –dry-run option:

rsync -ah --dry-run text.log [email protected]:/var/log

4.4. Transfer Multiple Files in a Directory

We can provide a list of files separated by a space to transfer multiple files:

rsync text.log text1.log text2.log [email protected]:/var/log

The -r option recursively copies all the files within the directory:

rsync -r /home/app/log [email protected]:/var/log

The * operator sends files matching a particular pattern:

rsync /home/app/log/app* [email protected]:/var/log

We can use the –include option and the –exclude option to include and exclude files from the list of files inside the directory. For example, to send all files from a directory excluding those starting with ‘R‘:

rsync --include '*' --exclude 'R*' text.log [email protected]:/urs/log

4.5. On-The-Fly Compression

We can reduce transfer time by using on-the-fly compression with the -z option. This will compress and decompress file at source and destination paths respectively:

rsync -z text.log [email protected]:/var/log

4.6. Delete Existing Files at Destination Path Before Transfer

The –delete option deletes existing files from the destination before transferring new files:

rsync --delete text.log [email protected]:/var/log

4.7. Automatically Remove Source Files After Transfer

Let’s say we want to take a backup of logs from the production server to a backup server. In this scenario, we would probably like to remove files from the production server after transferring all the files to our backup server.

rsync server provides the –remove-source-files option to remove source files after the transfer is complete:

rsync --remove-source-files text.log [email protected]:/var/log

5. Which Tool to Choose?

We can use both of these tools to transfer files between Linux machines. However, we should consider scp when the file size is small — it’s a simple copy tool and is widely available. In our day-to-day use, scp is the easy choice for small, one-off transfers.

We would benefit most from rsync when file size is big or when we need more complex and efficient synchronization. It’s also the better choice for recurring tasks like cron jobs and scripts.

6. Conclusion

In this article, we looked at two file transfer tools: scp and rsync.

We also looked at the differences between them, and which tools match certain use cases.

2 Comments
Oldest
Newest
Inline Feedbacks
View all comments
Comments are closed on this article!