Automate FTP Login and Scripting Using the Shell

1. Introduction

The File Transfer Protocol (FTP) along with its close relatives SSH File Transfer Protocol (SFTP) and File Transfer Protocol Secure (FTPS), all exist to enable data exchange between machines. To that end, we may require their use on a daily basis.

In this tutorial, we explore ways to automate logging in to an FTP, SFTP, or FTPS server. First, we start with an overview of a standard mechanism for auto-login. Next, we continue with the simple FTP protocol via a basic client. After that, an SFTP client takes the spotlight. Finally, we turn to a common client that supports FTP, FTPS, and SFTP, showing its abilities when it comes to automatic logins. As a bonus, we also cover two classic commands for file query and download, which support the protocols in question.

For security, we recommend the newest iteration of SFTP as implemented by OpenSSH. However, for the sake of completeness, we discuss clients that deal with each of the protocols.

We tested the code in this tutorial on Debian 11 (Bullseye) with GNU Bash 5.1.4. It should work in most POSIX-compliant environments.

2. .netrc

The .netrc file is a relatively universal way for auto-login to occur. Usually, its path is $HOME/.netrc, but we can reset that via the $NETRC environment variable or a command-line flag.

A .netrc file comprises a series of whitespace-separated token-parameter pairs:

$ cat $HOME/.netrc
default login anonymous
machine xost login baeldung password PASSWORD

Here, the first line defines the default username for login as anonymous, without a password. Such a line is optional, but there can only be one default.

On the second line, a login username of baeldung and a password of PASSWORD are specified for a machine that matches xost. In addition to password, we can use the account token to supply an optional secondary password.

Finally, there’s the macdef token, which defines a macro with its name just after it. Following the file format, the macro starts on the next line and continues until at least two consecutive newline characters are encountered:

$ cat $HOME/.netrc
default login anonymous
machine xost
  login baeldung
  password PASSWORD
  macdef init
    get tasks

machine gerganov.com login anonymous

In this case, the init macro runs a simple get to download the tasks file upon login automatically. While we can define any macro, only init runs directly on its own. Others have to be called manually by name via the FTP command.

Critically, due to the sensitive nature of the .netrc file, its mode should be 0600. This ensures that only the file creator can have read and write access to it. Still, storing a cleartext password doesn’t comply with best security practices.

3. ftp

Depending on the system, the regular ftp command might be an alias for different binaries. Regardless, the ftp package of a given Linux distribution commonly provides the standard ARPANET FTP functionality.

Here too, we assume ftp comes from the Debian ftp package. In turn, its binary is a chain of symbolic links that leads to netkit-ftp.

Unless a .netrc file match exists, ftp automatically prompts for a username and password when it starts:

$ ftp xost
Connected to xost.
Name: baeldung
331 Please specify the password.
Password:
230 Login successful.
ftp>

However, we can use -n to prevent both mechanisms, leaving us at the ftp> prompt without being logged in.

Instead of relying on the .netrc file, we can leverage this behavior of -n by piping login commands directly:

$ echo $'user baeldung PASSWORD\nget tasks' | ftp -n xost

In this case, we employ user to automatically login and download the tasks file with get. After this, tasks should be in our local working directory.

4. sftp

Although some SFTP clients support them via third-party implementations, both methods above shouldn’t work with the OpenSSH sftp client. The reason is simple: security isn’t compatible with the cleartext passwords in files or commands.

Still, we can use alternatives.

4.1. SSH Identity

Since sftp is based on the Secure Shell (SSH) protocol, we can leverage keys for authentication.

Specifically, we just need an authorized key without a password:

$ ssh-keygen -f xid -N ''
Generating public/private rsa key pair.
Your identification has been saved in xid
Your public key has been saved in xid.pub
The key fingerprint is:
[...]
$ cat xid.pub >> $HOME/.ssh/authorized_keys

Now, we simply run sftp from the same directory to perform an automatic and fairly safe login via the -i flag:

$ sftp -i xid baeldung@xost
$ sftp -i xid baeldung@xost
Connected to xost.
sftp>

At this point, we can run (S)FTP commands in the interactive session. However, we might want also to automate a script upon login.

4.2. Automatic Commands

By using the same means as with ssh, we can run commands automatically upon SFTP login.

Moreover, the -b flag of sftp supplies a file with commands to run when establishing an sftp session. In combination with process substitution, we can directly add a script:

$ sftp -i xid -b <(echo '
  get tasks
') xost

Here, we use the aforementioned -i flag to automatically login and then run the commands via -b.

4.3. expect

Combining the previous two methods, we can leverage the expect command for an automatic password login and passing instructions to a session:

$ expect <<EOI
spawn sftp baeldung@xost
expect "*password:"
send "PASSWORD\r"
expect "sftp>"
send "get tasks\r"
expect "sftp>"
send "bye\r"
EOI

In this case, we employ a here-string for the expect script. First, we spawn an sftp session to xost with user baeldung, expecting to get a password prompt that ends with password:. After sending the password, we wait for a command prompt. Finally, we get our tasks file and close the session.

5. lftp

As one of the most widespread and stable clients for FTP and its variants, we now turn to lftp. In fact, this client can use FTPS if the server is properly set up:

• FTP server installed
• TLS (SSL) certificate generated
• FTP server configured with the certificate

After these are in place, we can simply use the same steps as with regular FTP, apart from potentially supplying an ssl:key-file or ssl:cert-file in the lftp configuration:

• $HOME/.lftprc
• $HOME/.lftp/rc
• $HOME/.config/lftp/rc

Like other FTP clients, lftp also supports .netrc as a means for automatic login.

Let’s see what other options we have.

5.1. Auto-login Flags

The lftp client has a specific -u flag for providing login data:

$ lftp -u baeldung,PASSWORD xost
lftp baeldung@xost:~>

At this point, we’re logged in. Similarly, we can employ the –user and –password flags:

$ lftp --user baeldung --password PASSWORD xost
lftp baeldung@xost:~>

Notably, automatic login flags work for FTP and FTPS but don’t work for SFTP. Let’s check a method that applies to FTP, FTPS, and SFTP.

5.2. Universal Resource Identifier (URI)

An alternative way to supply the username and password is via the Universal Resource Identifier (URI):

$ lftp ftp://baeldung:PASSWORD@xost
lftp baeldung@xost:~>

The construction of a URI is fairly straightforward:

1. protocol prefix (ftp)
2. separator (://)
3. user (baeldung)
4. separator (:)
5. password (PASSWORD)
6. separator (@)
7. hostname or IP address (xost)

Since this is a standard URI, the method works by parsing its elements.

In fact, lftp can keep track of such URI addresses in so-called bookmark files:

• $HOME/.local/share/lftp/bookmarks
• $HOME/.lftp/bookmarks

To enable the feature, we add bmk:save-passwords true in the configuration. After that, we simply use the bookmark control commands within an lftp session to list, add, import, edit, and delete entries:

$ lftp -c 'bookmark add BOOKMARK_NAME ftp://USERNAME:[email protected]/path/to/subdirectory'
$ lftp -c 'open BOOKMARK_NAME && get REQUESTED_FILE'

Here, we add the BOOKMARK_NAME bookmark with the specified URI and then open it to get a REQUESTED_FILE automatically.

Since the password data is stored in cleartext, this breaks the best practices of password use. To partially work around this, in all cases, we can leverage the –env-password flag to make lftp get the passphrase from the $LFTP_PASSWORD environment variable.

6. wget and curl

As two versatile clients, curl and wget both support ways to perform an automatic FTP login just before an operation.

In fact, due to its ubiquity, .netrc is also supported by both clients.

6.1. curl

We can simply leverage the curl –user or -u flag for automatic FTP login:

$ curl --user baeldung:PASSWORD ftp://xost/

By supplying the colon-separated user credentials, the session is automatically established without further prompts.

6.2. wget

In addition to .netrc, wget supports its own .wgetrc file at /usr/local/etc/wgetrc or /etc/wgetrc.

In .wgetrc, we can specify several options:

• user – default username for any protocol
• password – default password for any protocol
• ftp_user (previously login) – default username for FTP
• ftp_password (previously passwd) – default password for FTP

Further, we can set and reset any of the options above with the respective wget flags:

• –user
• –password
• –ftp-user
• –ftp-password

Let’s see a simple wget example that performs a –recursive download of all files after an automatic login:

$ wget --recursive --ftp-user=baeldung --ftp-password=PASSWORD ftp://xost/

In all cases, the ftp* options take precedence.

7. Summary

In this article, we saw ways to log in to an FTP, FTPS, and SFTP server automatically.

In conclusion, while there are many methods to perform auto-login and run commands, some are insecure, while others require a specific client or protocol.