Automating LFTP Using Bash Scripts

1. Introduction

In this tutorial will explore options for automating lftp file transfers using bash scripts. lftp is a sophisticated FTP command-line client, supporting transfers in many protocols including FTP, FTPS, SFTP, HTTP, and HTTPS.

Sometimes, lftp comes pre-installed in Linux distributions. However, if that’s not the case, we can install it using the apt or yum commands, depending on our system. For Ubuntu/Debian:

$ sudo apt -y update && sudo apt install -y lftp

Or for Centos/RHEL:

$ sudo yum install lftp 

Next, let’s look at some of the options we have available for lftp automation.

2. Basic File Transfer Script

First, let’s write a basic file transfer script with lftp:

#!/bin/bash

# Define variables
FTP_HOST="ftp.example.com"
FTP_USER="username"
FTP_PASS="password"
LOCAL_FILE="/path/to/local/file"
REMOTE_FILE="/path/to/remote/file"

# Upload file to FTP server
lftp -u $FTP_USER,$FTP_PASS ftp://$FTP_HOST -e "put $LOCAL_FILE -o $REMOTE_FILE; bye"

# Download file from FTP server
lftp -u $FTP_USER,$FTP_PASS ftp://$FTP_HOST -e "get $REMOTE_FILE -o $LOCAL_FILE; bye"

First, we define the connection endpoints and credentials. Then, we include commands to both upload and download a file using lftp. Note that the -e parameter allows multiple commands chained with semicolons.

3. Scripting Through Here Documents

When we need to transfer multiple files within the same LFTP session, we can use bash here documents. Let’s modify our script to achieve this:

#!/bin/bash

# Define variables
FTP_HOST="ftp.example.com"
FTP_USER="username"
FTP_PASS="password"
LOCAL_FILE="/path/to/local/file"
REMOTE_FILE="/path/to/remote/file"

# Use a here-document to send commands to lftp
lftp -u $FTP_USER,$FTP_PASS ftp://$FTP_HOST <<EOF
put $LOCAL_FILE -o $REMOTE_FILE
get $REMOTE_FILE -o $LOCAL_FILE
bye
EOF

Now, upload and download are done in a single lftp command execution. All strings between “<<EOF” and “EOF” will be piped to the lftp command.

4. Using lftp Script Files

The lftp utility can also read the command chain in a text file using the “-f” parameter. This is more readable for longer command sequences.

First, we’ll create an lftp script file, in this example, named “lftp_script.txt“:

# lftp_script.txt
put /path/to/local/file -o /path/to/remote/file
get /path/to/remote/file -o /path/to/local/file
bye

As with shell scripts, anything on a line after a “#” is a comment.

Now, we can create the bash script using this script file:

#!/bin/bash

# Define variables
FTP_HOST="ftp.example.com"
FTP_USER="username"
FTP_PASS="password"

# Run the lftp script
lftp -u $FTP_USER,$FTP_PASS -f lftp_script.txt

This option abstracts the lftp-specific commands from the bash ones. So, the result gets easier to maintain. Also, we can have different access permissions for each file.

5. Pseudo-Interactive Sessions Using expect

What if we must emulate any user interaction, such as confirming prompts, taking alternative paths, or handling errors?

That’s when the expect utility comes in handy.

To install it, in case we don’t have it yet, we can use, for Ubuntu/Debian:

$ sudo apt -y update && sudo apt install -y expect 

Or for CentOS/RHEL:

$ sudo yum install expect

We can write an expect script file or use it with a here document to control the lftp execution. So, let’s create an expect script file, named “expect_script.txt”:

set timeout 60
set prompt "lftp *"
set FTP_HOST "$::env(FTP_HOST)"
set FTP_USER "$::env(FTP_USER)"
set FTP_PASS "$::env(FTP_PASS)"
set LOCAL_FILE "$::env(LOCAL_FILE)"
set REMOTE_FILE "$::env(REMOTE_FILE)"
spawn lftp -u $FTP_USER,$FTP_PASS ftp://$FTP_HOST

# Uploads file
expect {
   $prompt { $prompt send "put $LOCAL_FILE -o $REMOTE_FILE\r" }
   timeout { puts "Timed out"; exit 1}
}

# Downloads file
expect {
   $prompt {send "get $REMOTE_FILE -o $LOCAL_FILE\r" } 
   timeout { puts "Timed out"; exit 1} 
}

send "bye\r"
expect eof

Here, our script sets a 60-second timeout, a specific lftp prompt, and the variables to use from the environment. Then, it spawns an lftp process and tries to transfer files until the timeout expires.

Then, we can write the bash script invoking expect:

#!/bin/bash

# Define variables
export FTP_HOST="ftp.example.com"
export FTP_USER="username"
export FTP_PASS="password"
export LOCAL_FILE="/path/to/local/file"
export REMOTE_FILE="/path/to/remote/file"

# Use expect to automate login and file transfer
expect -f expect_script.txt

The expect command spawns lftp and waits for the required prompts before sending commands. Also, we can define timeouts and alternative actions in case the expect command does not receive the prompts.

6. Error Handling

As with most command-line utilities, lftp returns differently when something fails. That way, we can check its results on the bash script:

#!/bin/bash

# Define variables
FTP_HOST="ftp.example.com"
FTP_USER="username"
FTP_PASS="password"
LOCAL_FILE="/path/to/local/file"
REMOTE_FILE="/path/to/remote/file"
LOG_FILE="/path/to/logfile.log"

# Run lftp and capture output
lftp -u $FTP_USER,$FTP_PASS ftp://$FTP_HOST -e "put $LOCAL_FILE -o $REMOTE_FILE; bye" >> $LOG_FILE 2>&1

# Check if the transfer was successful
if [ $? -eq 0 ]; then
    echo "File transfer successful" >> $LOG_FILE
else
    echo "File transfer failed" >> $LOG_FILE
fi

The if condition checks whether the command executed correctly (return code 0) or not (return code other than 0). Also, this script saves all output in a log file. We can also check result codes if we’re using expect.

7. Scheduling With Cron Jobs

As with batch routines, sometimes, we may want to run lftp transfers without user intervention at specific times. To do so, we can use the cron process scheduler.

To edit the Cron schedule for a specific user, we use:

crontab -e -u <user>

Now, to run the file transfer script every day at 3:15 am, we add a line at the end of the file:

15 3 * * * /path/to/your/script.sh

The script file path and name should be adjusted accordingly.

8. Conclusion

In this article, we’ve shown several methods for automating lftp file transfers using bash scripts on Linux, from basic to more advanced scenarios.

By using these methods, we can fulfill many batch file transfer tasks, using any lftp-supported protocols. Finally, we can also automate the lftp login so that the script files don’t need clear text passwords.