Request a File Without Saving in Linux

1. Overview

Consider a situation where we want to make a dummy request to the server, but we don’t want to download the resources. We might do this to check if the server is up or to warm up the cache, or for some other reason.

The wget and curl commands can be used to request a file and avoid saving the data in response.

2. Using the wget Command

GNU wget is an open-source, non-interactive command-line utility for downloading files from the web. With wget, we can download files using HTTP, HTTPS, and FTP protocols.

wget offers several options that allow us to download multiple files, resume downloads, bandwidth throttling, recursive downloads, background downloads, mirror sites, and more.

Now let’s see how we can use wget to avoid saving the file.

2.1. Output Redirection

In Linux, the commands take input from the terminal and return the output back to the terminal. Output redirection helps us divert the output to places other than the default terminal. We can redirect the output to a file, directory, or as input to another command.

wget uses the concept of output redirection to redirect output to a directory and save the file with a different name. In general, we can redirect any output to /dev/null to avoid saving the file to the machine:

$ wget -O /dev/null https://github.com/Baeldung/spring-security-oauth/archive/refs/heads/master.zip 
--2022-02-09 17:49:12--  https://github.com/Baeldung/spring-security-oauth/archive/refs/heads/master.zip
Resolving github.com (github.com)... 140.82.113.3
Connecting to github.com (github.com)|140.82.113.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/Baeldung/spring-security-oauth/zip/refs/heads/master [following]
--2022-02-09 17:49:13--  https://codeload.github.com/Baeldung/spring-security-oauth/zip/refs/heads/master
Resolving codeload.github.com (codeload.github.com)... 140.82.114.9
Connecting to codeload.github.com (codeload.github.com)|140.82.114.9|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: '/dev/null'

/dev/null             [ <=>                        ]   1.49M  --.-KB/s    in 0.1s    

2022-02-09 17:49:13 (10.1 MB/s) - '/dev/null' saved [1564839]

In the above command, the -O option redirects the output to a file. We are redirecting the output to a special file, /dev/null. It is a standard file that discards all the data written into it, and at the same time, it returns the success for the write operation performed.

Let’s verify that the wget command did not save the master.zip file locally:

$ ls -lah
total 0
drwxr-xr-x. 2 root root  6 Feb  9 17:36 .
drwxr-xr-x. 7 root root 97 Feb  9 17:36 ..

By default, the wget command redirects the output to the current shell:

$ wget -O- https://github.com/Baeldung/spring-security-oauth/archive/refs/heads/master.zip

This time, the content of the master.zip file will display on the terminal, and the file will not be saved.

In the output above, the wget command output was printed on the terminal. It includes a bunch of useful network information. But what if we wish to discard that output as well. We can use the -q option for this purpose:

$ wget -qO /dev/null https://github.com/Baeldung/spring-security-oauth/archive/refs/heads/master.zip

We use -q flag to quiet the wget command output.

2.2. Using the –delete-after Flag

A naive approach to request a file without saving is that we first download that file and remove it just after the download is complete. The –delete-after flag of wget command does this out-of-the-box:

$ wget --delete-after https://github.com/Baeldung/spring-security-oauth/archive/refs/heads/master.zip
--2022-02-09 17:54:49--  https://github.com/Baeldung/spring-security-oauth/archive/refs/heads/master.zip
Resolving github.com (github.com)... 140.82.113.3
Connecting to github.com (github.com)|140.82.113.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/Baeldung/spring-security-oauth/zip/refs/heads/master [following]
--2022-02-09 17:54:49--  https://codeload.github.com/Baeldung/spring-security-oauth/zip/refs/heads/master
Resolving codeload.github.com (codeload.github.com)... 140.82.114.10
Connecting to codeload.github.com (codeload.github.com)|140.82.114.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Saving to: 'master.zip.tmp'

master.zip.tmp       [ <=>                        ]   1.49M  --.-KB/s    in 0.1s    

2022-02-09 17:54:50 (10.0 MB/s) - 'master.zip.tmp' saved [1564839]

Removing master.zip.tmp.

Here, the requested resource, master.zip, was first downloaded as a temporary file. It then removes the downloaded file. –delete-after flag does this implicitly.

2.3. Using the –spider Flag

The –spider option in wget command just check if the resource is available or not without downloading it:

$ wget --spider https://github.com/Baeldung/spring-security-oauth/archive/refs/heads/master.zip
Spider mode enabled. Check if remote file exists.
--2022-02-10 16:48:21--  https://github.com/Baeldung/spring-security-oauth/archive/refs/heads/master.zip
Resolving github.com (github.com)... 140.82.112.4
Connecting to github.com (github.com)|140.82.112.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/Baeldung/spring-security-oauth/zip/refs/heads/master [following]
Spider mode enabled. Check if remote file exists.
--2022-02-10 16:48:21--  https://codeload.github.com/Baeldung/spring-security-oauth/zip/refs/heads/master
Resolving codeload.github.com (codeload.github.com)... 140.82.112.10
Connecting to codeload.github.com (codeload.github.com)|140.82.112.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/zip]
Remote file exists.

Unlike the other two approaches that we have discussed so far, this approach does not download the resource on the machine. Hence this is the most optimized solution among all.

3. Using the curl Command

curl is a command-line utility for the libcurl library. It helps in the transfer of data between the client and the server.

Let’s use the curl command to check if the resource is available or not:

$ curl https://github.com/Baeldung/spring-security-oauth/archive/refs/heads/master.zip > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   146  100   146    0     0   1327      0 --:--:-- --:--:-- --:--:--  1315

Just like wget command, we have used the redirection operator to redirect the downloaded content to /dev/null. At last, we can verify that no file was downloaded by using the ls command.

4. Conclusion

In this tutorial, we looked at different approaches to request a file without saving it locally. By using the output redirection mechanism with the curl and the wget commands, we can discard the output. Moreover, we’ve used a couple of options supported by the wget command to get the job done.