Guide to Downloading URLs Listed in a File Using cURL

1. Overview

cURL is a popular software that provides a command line tool for transferring data using various network protocols.

In this tutorial, we’ll learn how to use the curl command to download a set of URLs listed in a file.

2. Sample File

Let’s start by creating the file-with-urls.txt file to contain a list of file URLs:

$ echo "file://$HOME/file1" >> $HOME/file-with-urls.txt; \
> echo "file://$HOME/file2" >> $HOME/file-with-urls.txt; \
> echo "file://$HOME/file3" >> $HOME/file-with-urls.txt;

We can note that we’ve specified the $HOME variable to use the user’s home directory as a parent directory for all the files.

Next, we know that the files file1, file2, and file3 don’t exist yet. So, let’s go ahead and create these files with some sample textual data:

$ echo "content for file1" >> $HOME/file1; \
> echo "content for file2" >> $HOME/file2; \
> echo "content for file3" >> $HOME/file3;

Finally, let’s verify the content within these sample files while using the -n option of the tail command so that we can see both the filenames and the content:

$ tail -n +1 file-with-urls.txt file1 file2 file3
==> file-with-urls.txt <==
file:///Users/tavasthi/file1
file:///Users/tavasthi/file2
file:///Users/tavasthi/file3

==> file1 <==
content for file1
content for file1

==> file2 <==
content for file2

==> file3 <==
content for file3

It looks like we’ve got this right!

3. curl and xargs

First, let’s see how we can download a single file using the -O option of the curl command:

$ curl -O file://$HOME/file1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    18  100    18    0     0  32667      0 --:--:-- --:--:-- --:--:-- 32667
$ cat file1
content for file1

Next, our problem boils down to how we can read the file-with-urls.txt file one line at a time and send it to the curl command for downloading one file at a time. The -n option available with the xargs command solves this use case.

So, let’s go ahead and use the xargs command for passing a single file URL as an argument to the curl command for download:

$ xargs -n 1 curl -O < $HOME/file-with-urls.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    18  100    18    0     0  23622      0 --:--:-- --:--:-- --:--:-- 23622
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    18  100    18    0     0  32667      0 --:--:-- --:--:-- --:--:-- 32667
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    18  100    18    0     0  40449      0 --:--:-- --:--:-- --:--:-- 40449

Finally, let’s verify that the files are downloaded correctly:

$ tail -n +1 ./file1 ./file2 ./file3
==> ./file1 <==
content for file1

==> ./file2 <==
content for file2

==> ./file3 <==
content for file3

We can see that the downloaded files are exactly what we were looking for!

4. curl With while Loop

We can also use the while loop for solving the problem of reading file-with-urls.txt line by line and then pass a single file URL as an argument to the curl command.

Let’s go ahead and write a one-liner solution for downloading all the files listed in the file-with-urls.txt file:

$ while read -r line; do curl -O $line; done < $HOME/file-with-urls.txt

We must note that the call to the read command gives us one file URL in the line variable at a time. Then, the curl command within the loop executes iteratively to download each file.

Lastly, let’s verify the downloaded files:

$ tail -n +1 ./file1 ./file2 ./file3
==> ./file1 <==
content for file1

==> ./file2 <==
content for file2

==> ./file3 <==
content for file3

The result looks perfect!

5. wget as an Alternative

In this section, let’s explore the alternative approach of using the wget command with the -i option to solve the same use case.

First, let’s see if we can get it working for downloading the files listed in the file-with-urls.txt file:

$ wget -i $HOME/file-with-urls.txt
/Users/tavasthi/file-with-urls.txt: Invalid URL file:///Users/tavasthi/file1: Unsupported scheme ‘file’
/Users/tavasthi/file-with-urls.txt: Invalid URL file:///Users/tavasthi/file2: Unsupported scheme ‘file’
/Users/tavasthi/file-with-urls.txt: Invalid URL file:///Users/tavasthi/file3: Unsupported scheme ‘file’
No URLs found in /Users/tavasthi/file-with-urls.txt.

It looks like the attempt failed, and the error message indicates that the wget command doesn’t support the file scheme. This is a significant limitation of wget compared to the curl command that supports this scheme.

Moving on, let’s keep in mind that the wget command supports limited schemes such as http, https, and ftp. Henceforth, let’s go ahead and use a different sample file that contains a list of file URLs with the https scheme:

$ cat $HOME/file-with-http-urls.txt
https://filesamples.com/samples/document/txt/sample1.txt
https://filesamples.com/samples/document/txt/sample2.txt

Finally, let’s download these files using wget with the -i option:

$ wget -i $HOME/file-with-http-urls.txt
--2022-12-22 08:23:44--  https://filesamples.com/samples/document/txt/sample1.txt
Resolving filesamples.com (filesamples.com)... 172.67.178.244, 104.21.17.252
Connecting to filesamples.com (filesamples.com)|172.67.178.244|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 607 [text/plain]
Saving to: ‘sample1.txt’

sample1.txt                                100%[========================================================================================>]     607  --.-KB/s    in 0s

2022-12-22 08:23:45 (2.64 MB/s) - ‘sample1.txt’ saved [607/607]

--2022-12-22 08:23:45--  https://filesamples.com/samples/document/txt/sample2.txt
Reusing existing connection to filesamples.com:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘sample2.txt’

sample2.txt                                    [ <=>                                                                                     ]   2.79K  --.-KB/s    in 0s

2022-12-22 08:23:45 (9.81 MB/s) - ‘sample2.txt’ saved [2859]

FINISHED --2022-12-22 08:23:45--
Total wall clock time: 1.2s
Downloaded: 2 files, 3.4K in 0s (6.65 MB/s)

We can see that this time, the download worked as expected. Additionally, we must realize that for the supported schemes, using the wget was more straightforward than using the curl command.

6. Conclusion

In this article, we learned a few ways to use the curl command for downloading a set of files listed in a file. Additionally, we learned the alternative of the wget command to solve the exact use case with ease.