Baeldung Pro – Linux – NPI EA (cat = Baeldung on Linux)
announcement - icon

Learn through the super-clean Baeldung Pro experience:

>> Membership and Baeldung Pro.

No ads, dark-mode and 6 months free of IntelliJ Idea Ultimate to start with.

1. Overview

In this tutorial, we’ll learn how to display copy progress and speed on Linux.

2. Single File

rsync is a file-copying tool that works for remote as well as local copies. It isn’t always installed by default, but it’s a popular tool and can be installed from standard repositories.

Let’s use rsync to copy a single file while displaying progress. The destination can be a file or a directory:

rsync --progress /path/to/source-file /path/to/destination
source-file
    264,601,600  25%  126.22MB/s    0:00:06

We see the number of bytes copied so far, the progress percentage, speed, and time remaining.

3. Directory

Let’s copy a directory using rsync while displaying the progress:

rsync -r --progress /path/to/source-dir /path/to/destination-dir
sending incremental file list
created directory /path/to/destination-dir
source-dir/
source-dir/1
    104,857,600 100%  261.70MB/s    0:00:00 (xfr#1, to-chk=8/10)
source-dir/2
    104,857,600 100%  102.46MB/s    0:00:00 (xfr#2, to-chk=7/10)
source-dir/3
    104,857,600 100%   58.11MB/s    0:00:01 (xfr#3, to-chk=6/10)
...

We can see that for recursive copy, rsync will show progress info for each file separately. What if we want to see overall progress instead?

Let’s use a different option to see overall progress:

rsync -r --info=progress2 /path/to/source-dir /path/to/destination-dir
    423,297,024  44%  134.64MB/s    0:00:03  (xfr#4, to-chk=5/10)

This command will copy source-dir inside destination-dir. We can see the number of bytes copied, overall completion percentage, the total rate of transfer, and the time remaining. xfr#4 means four files were transferred so far, to-chk=5/10 means that out of ten files, five remain to be checked by rsync to see if they’re up to date.

We should be aware that putting a slash at the end of source-dir will make rsync behave differently. This will copy the contents of source-dir, not source-dir itself:

rsync -r --info=progress2 /path/to/source-dir/ /path/to/destination-dir

4. Other Methods

We might not think of using curl, the URL transfer tool, to copy a file locally. It’s possible, but for only one file at a time:

curl -o /path/to/destination-file file:///path/to/source-file
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 17 1000M   17  173M    0     0   518M      0  0:00:01 --:--:--  0:00:01  518M

Let’s use progress, a tool that will display the progress of basic commands already running, like cp, mv, dd, tar or gzip:

progress -M
[ 2498] cp /path/to/source-file
        80.7% (806.9 MiB / 1000 MiB) 245.0 MiB/s

Finally, let’s try lsof, which can also be used to get information about a copy that’s already in progress. We can monitor the changes using watch:

watch lsof /path/to/destination
COMMAND  PID USER   FD   TYPE DEVICE  SIZE/OFF   NODE NAME
cp      2197 root    4w   REG    8,2 650133504 539646 /path/to/destination

4. The pv Command

The pv (Pipe Viewer) command is a versatile tool for monitoring the progress of data through a redirection or pipeline in Linux. Whether we’re copying files, transferring data over networks, or working with block devices, pv provides valuable feedback about our operation’s progress.

Next, let’s understand how pv works through some examples.

First, let’s copy a file using pv:

$ pv /path/to/source-file > /path/to/destination
2.18GiB 0:00:02 [2.15GiB/s] [=========================>                ] 64% ETA 0:00:02

As we can see, using pv, the copy syntax looks a bit different. We transferred the file through a redirection instead of copying, and pv monitored the transfer progress.

The pv command provides the –progress option to show a progress bar. This option is enabled by default. Therefore, we can see the progress bar by simply running pv without any options.

Next, let’s understand the report in pv‘s default output:

  • 2.18GiB – Total amount of data has been transferred so far
  • 0:00:02 – A timer to display the total elapsed time that pv has been running for
  • [2.15GiB/s] – The current rate of data transfer
  • [==> ] 64% – A progress bar with a percentage indicating the transfer progress
  • ETA 0:00:02 – The ETA countdown, which estimates how long the transfer will take, based on current transfer rates and the total data size

It’s worth mentioning that the pv command offers a rich set of options that allow us to control the transfer or customize the progress report flexibly.

For example, we can use the -L option to limit the transfer rate:

$ pv -L 50m /path/to/source-file > /path/to/destination
700MiB 0:00:16 [47.8MiB/s] [======>                                     ] 15% ETA 0:01:49

The above example limited the transfer rate to below 50MiB/s.

If we leverage a pipe to perform file transfer over the network, pv can monitor network transfers. For example, we can use pv to report the progress of sending a local file to a remote server over SSH:

$ pv /path/to/source-file | ssh user@remove-server "cat > /path/to/destination"
10.0MiB 0:00:02 [5.22MiB/s] [======>                                    ]  11% ETA 0:00:12

Similarly, we can monitor the progress of pulling a file from a remote server to the local machine over SSH:

$ ssh user@remove-server "cat /path/to/source-file" | pv > /path/to/destination
12.0MiB 0:00:02 [7.25MiB/s] [========>                                  ]  14% ETA 0:00:10

As we can see, since pv monitors the data transfer through a pipe, it’s capable of monitoring arbitrary data flows through pipes.

5. Other Methods

We might not think of using curl, the URL transfer tool, to copy a file locally. It’s possible, but for only one file at a time:

curl -o /path/to/destination-file file:///path/to/source-file
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 17 1000M   17  173M    0     0   518M      0  0:00:01 --:--:--  0:00:01  518M

Let’s use progress, a tool that will display the progress of basic commands already running, like cp, mv, dd, tar or gzip:

progress -M
[ 2498] cp /path/to/source-file
        80.7% (806.9 MiB / 1000 MiB) 245.0 MiB/s

Finally, let’s try lsof, which can also be used to get information about a copy that’s already in progress. We can monitor the changes using watch:

watch lsof /path/to/destination
COMMAND  PID USER   FD   TYPE DEVICE  SIZE/OFF   NODE NAME
cp      2197 root    4w   REG    8,2 650133504 539646 /path/to/destination

6. Conclusion

In this quick article, we learned how to see progress while copying files and directories on Linux.