How to Monitor Running rsync Processes

1. Overview

Monitoring running rsync processes can be crucial for system administrators. Whether it’s ensuring data integrity, optimizing performance, or troubleshooting issues, knowing what rsync is doing can provide valuable insights.

In this tutorial, we’ll explore methods to monitor already running rsync processes, especially those initiated via cron jobs without predefined verbosity and logging options. We’ll delve into using tools like strace, ps, and lsof to see which files and folders are synced.

2. Introduction to rsync

rsync is a powerful tool for synchronizing files and directories between different systems. It’s often used in backup solutions, data mirroring, and migration tasks due to its efficiency and flexibility.

When run with appropriate options, rsync can provide detailed logs and verbose output, but this isn’t always the case. Sometimes, especially with cron jobs, rsync runs in the background with minimal logging. Monitoring these processes requires additional tools and techniques.

2.1. Verbosity and Logging Options in rsync

rsync offers various logging and verbosity options that can provide us with detailed information about the files and directories being synchronized. These options are useful when we want to track rsync‘s activities or troubleshoot synchronization issues.

The -v (verbose) flag instructs rsync to increase the level of detail in its output. We can use it multiple times to get more detailed information, for example:

• -v: displays the names of files being transferred
• -vv: shows additional details, such as file attributes
• -vvv: provides the most detailed information, including protocol-level details

We can also create a log file for rsync‘s activities using the –log-file option. This allows us to capture the output of rsync, even if it’s running in the background or via a cron job. For example:

$ rsync -av --log-file=/var/log/rsync.log /source/ /destination/

This command logs the synchronization details to /var/log/rsync.log. Combined with verbosity options, it can give us a detailed record of all the files transferred, making it easier to track what rsync is doing at any given time.

3. Using ps to Identify rsync Processes

The ps command is the first tool in our monitoring toolkit. It helps us identify running rsync processes and gather basic information about them.

3.1. Listing Processes

Let’s start by listing all running rsync processes:

$ ps aux | grep rsync

This command lists all processes and filters the results to show only those related to rsync. The output includes the process ID (PID), user, CPU usage, and more.

3.2. Understanding the Output

The ps aux command provides a lot of information, but for monitoring purposes, we’re mainly interested in the PID and command line. Here’s a sample output:

$ ps aux | grep rsync
user     12345  0.5  0.1  10520  2048 ?        Ss   12:34   0:00 rsync -avz /source/ /destination/

In this case, the PID is 12345. With the PID, we can delve deeper into what the process is doing using other tools.

4. Tracking File Operations with strace

strace is a powerful diagnostic, debugging, and instructional tool. It monitors system calls made by a running process and signals received by the process.

4.1. Attaching strace to an rsync Process

To attach strace to an already running rsync process, we’ll use the PID obtained with the help of ps aux:

$ sudo strace -p 12345 -e trace=file

We need to replace 12345 with the actual PID of the rsync process. The -e trace=file option limits the output to file-related system calls, making it easier to see which files rsync is accessing.

4.2. Interpreting strace Output

The output will show various system calls related to file operations, such as open, read, write, and close. For example:

$ sudo strace -p 12345 -e trace=file
...
open("/source/file1.txt", O_RDONLY) = 3
read(3, "data", 1024) = 1024
write(4, "data", 1024) = 1024
close(3) = 0
...

This snippet shows that rsync opened a file, read some data, wrote the data, and then closed the file. By monitoring these calls, we can see exactly what rsync is doing in real-time.

5. Using lsof to List Open Files

lsof is another useful tool for monitoring rsync processes. It lists all open files and the processes that opened them.

5.1. Running lsof

To list open files for an rsync process, we can use:

$ lsof -c rsync -s -a -d1,3

The -c option instructs lsof to list the files opened by all processes executing the command that begins with the string “rsync”, while the -s option is used to display the file size at all times. This is useful to see how the destination file size changes as rsync synchronizes it with the source.

The -a option instructs lsof to AND all the list selection options. Finally, the option -d1,3 is used to limit the display to file descriptors of types 1 and 3.

5.2. Analyzing lsof Output

The lsof output includes several columns, such as:

• COMMAND: the command that opened the file
• PID: the process ID
• USER: the user who owns the process
• FD: the file descriptor
• TYPE: the type of file (e.g., regular file, directory)
• SIZE: the size of the file in bytes
• NAME: the name of the file

Here’s a sample output:

$ lsof -c rsync -s -a -d1,3
COMMAND  PID    USER   FD   TYPE DEVICE SIZE     NODE NAME
rsync    12345  user   3r   REG  8,1    4096         12345 /source/file1.txt
rsync    12345  user   1u   REG  8,1    4096         67890 /destination/.file1.txt.btRhdu
...

This shows that rsync has opened /source/file1.txt for reading and /destination/.file1.txt.btRhdu for writing. The latter is a temporary file that rsync uses to write the file before it is renamed to the final name after the write operation is concluded. In essence, we can use this information to understand which files are being synchronized.

6. Combining Tools for Comprehensive Monitoring

Using ps, strace, and lsof together provides a comprehensive view of rsync‘s activities. Given that, each tool offers unique insights, and combining them allows us to monitor rsync processes effectively. Overall, we can utilize the following workflow:

1. Identify rsync Processes: use ps aux | grep rsync to find running rsync processes and their PIDs
2. Trace System Calls: attach strace to the rsync process to monitor file-related system calls and see real-time file operations
3. List Open Files: use lsof to list all open files for the rsync process; this provides a snapshot of files currently accessed

By following this workflow, we can monitor rsync processes comprehensively, even if they were started without detailed logging options.

7. Conclusion

Tools like ps, strace, and lsof enable us to gain insights into rsync’s activities, even when it runs in the background without detailed logging.

Using ps, we can identify running rsync processes and their PIDs. With strace, we can monitor file-related system calls in real-time, and lsof provide a snapshot of all open files. Combining these tools gives us a comprehensive view of rsync‘s operations.

In essence, while rsync itself is a powerful tool, knowing how to monitor its processes effectively enhances our ability to manage and maintain our systems. Moreover, by leveraging the capabilities of ps, strace, and lsof, we can keep a close eye on rsync, ensuring it performs its synchronization tasks correctly and efficiently.