Using Proxy With wget

1. Overview

wget is a tool for downloading files from the internet via the Linux command line. However, sometimes when downloading, we might need a proxy to access certain sites or maintain our privacy. Fortunately, we can configure wget to use a proxy via protocols such as HTTP, HTTPS, and FTP.

In this tutorial, we’ll learn how to set up a proxy with wget by editing the wget configuration file, setting environment variables, and using other methods.

2. Why Use Proxy With wget

We often use a proxy to enhance our downloading experience by masking our real IP address and adding a layer of privacy and anonymity. When using a proxy while downloading files using wget, our location remains hidden, which is ideal if we value online privacy.

Additionally, proxies can improve download efficiency. For example, some proxy servers cache frequently requested content, meaning that if we download the same file multiple times, the proxy may serve a cached version, thereby speeding up the process.

2.1. Installation

Before getting started, let’s ensure that wget is installed on our system. Most Linux distributions come with wget pre-installed, but if it’s not already present, we can install it using the default package manager. For example, to install wget on Debian/Ubuntu Linux, we can use apt:

$ sudo apt install wget

Moreover, we can confirm the installation by executing wget –version in the terminal.

3. Setting a Proxy With wget

wget offers several ways to set a temporary or permanent proxy, both for system-wide use or for a single user. For example, we can edit the configuration file if we want wget to always use a proxy, or we can specify it directly on the command line using the -e option.

3.1. Using Command-Line Options

The most straightforward way to use a proxy with wget is to specify it directly in the command using the -e option. This method is useful when we need to set a proxy for a single wget command without altering global settings.

Let’s download a file using a specific proxy:

$ wget -e use_proxy=yes -e http_proxy=http://yourproxyaddress:port http://example.com/file.zip

Here, we use -e use_proxy=yes to enable wget to use a proxy, and -e http_proxy= to set the specific proxy address and port.

3.2. Using Environment Variables

The simplest way to set up a proxy for wget is by defining environment variables in the shell configuration file (e.g., .bashrc or .bash_profile). These key variables are http_proxy for HTTP requests, https_proxy for HTTPS requests, and ftp_proxy for FTP requests.

To begin, we need to open the shell configuration file using a text editor, like nano:

$ nano ~/.bashrc

Then, we need to add the following lines in the file:

export http_proxy="http://proxy_server:port"
export https_proxy="http://proxy_server:port"
export ftp_proxy="http://proxy_server:port"

Also, we must make sure to replace proxy_server and port with our actual proxy details.

After editing the file, we need to apply the changes by restarting our terminal or sourcing the file with the source command:

$ source ~/.bashrc

To confirm that our proxy configuration is working correctly, we can attempt a download:

$ wget http://example.com

A successful download indicates that the proxy is configured correctly. For further verification, we can use a site like ipinfo to check the IP address seen by the server, ensuring it matches the proxy’s IP.

What if we want to test proxy without altering our permanent settings? In that case, we can set the environment variables directly in the terminal:

$ export http_proxy="http://proxy_server:port"
$ export https_proxy="http://proxy_server:port"
$ export ftp_proxy="http://proxy_server:port"

Also, if our proxy requires authentication, we include our credentials in the URL format:

$ export http_proxy="http://username:password@proxy_server:port"

Further, we need to replace username, password, proxy_server, and port with the actual values. It’s also important to note that environment variables are case-sensitive, and wget expects lowercase names.

3.3. Editing the wget Configuration File (wgetrc)

For a more permanent solution, we need to configure wget to always use a proxy by modifying its configuration file. It’s located at /etc/wgetrc for system-wide settings or ~/.wgetrc for user-specific settings. By editing this file, we can establish persistent proxy configurations without relying solely on shell variables.

For example, to set the proxy for all users, we need to edit /etc/wgetrc with administrative privileges:

$ sudo nano /etc/wgetrc

We then add or modify the following lines in the confirmation file to apply the setting system-wide:

http_proxy = http://proxy_server:port
https_proxy = http://proxy_server:port
ftp_proxy = http://proxy_server:port
use_proxy = on

Furthermore, for our individual settings, we can edit our personal ~/.wgetrc file:

$ sudo nano ~/.wgetrc

Next, we need to add the same lines as above to configure our proxy without requiring administrative rights.

Also, if we work in an environment where different proxy settings are needed for different projects, we might want to create separate configuration files or use conditional settings in our .wgetrc.

4. Conclusion

In this article, we learned how to integrate a proxy with wget using multiple methods—from setting environment variables and editing the wget configuration file, to using command-line options.

Each approach offers unique benefits, whether we need a temporary fix or a permanent configuration. Additionally, we learned how to incorporate authentication details when required, ensuring that our proxy setup remains secure and effective.

By applying these techniques, we can mask the system IP address, improve download performance, and maintain greater privacy while accessing the web.