Resolving the Reverse Tunnel Remote port forwarding failed for listen port SSH Error

1. Introduction

As the ubiquitous method to facilitate and secure connections between remote systems, the Secure Shell (SSH) protocol supports multiple ways to handle point-to-point communication. Furthermore, one of the most useful features in this regard is remote port forwarding. In one of its basic forms, the latter enables going through Network Address Translation (NAT) filters and similar firewall restrictions.

In this tutorial, we talk about remote SSH port forwarding or reverse tunnels. First, we understand the problems that these features solve. Next, we go through the prerequisites and process of establishing a reverse tunnel. After that, we explore problems that might result from different aspects of SSH remote port forwarding, i.e., reverse tunnels.

We tested the code in this tutorial on Debian 12 (Bookworm) with GNU Bash 5.2.15. Unless otherwise specified, it should work in most POSIX-compliant environments.

2. Remote or Reverse Tunneling and Client Protection

To begin with, let’s understand the concept of remote or reverse tunneling. In essence, the idea is to use SSH from a protected client to establish a permanent tunnel to a remote server, enabling traffic in both directions.

There can be different kinds of client protection in this context:

• behind a router
• NAT
• firewall
• Virtual Private Network (VPN)

Basically, we can use this technique to gain access to the client side and manage any network resource that the client can control, all from a remote server that normally can’t directly connect to that client. In addition, local client ports can be forwarded to the remote server side.

This idea has two main names: remote tunneling and reverse tunneling, depending on the way it’s applied. In all cases, it provides port forwarding.

3. SSH Remote Port Forwarding

Remote port forwarding or reverse tunneling is a special case of remote tunneling, which exposes a local service or application to a remote server even when the local machine is protected in any of the ways mentioned earlier.

3.1. Requirements

To employ remote port forwarding with SSH, we need several prerequisites to be in place:

• SSH client on the client machine
• SSH server on the server machine
• network connectivity and visibility from client to server

In this case, we use xost as the server.

3.2. SSH Client -R Flag

When it comes to most SSH clients, the way to perform remote port forwarding is the -R flag, which takes a colon-separated list as its value. The latter contains four fields:

1. (optional, default localhost) remote_server_bind_address: address to use on the remote server for the reverse tunnel
2. remote_server_bind_port: port to use on the remote server for the reverse tunnel connection
3. client_network_host: address that the client has network access to
4. client_network_host_port: port at client_network_host

The SSH argument is the remote server that should be the opposite end of the tunnel.

In essence, connections to [remote_server_bind_address]:remote_server_bind_port on the remote server (provided as an argument) should be forwarded to the client network at client_network_host:local_port. Notably, both address-port combinations can be sockets instead. Without a remote_server_bind_address, connections only work for loopback interfaces, while an empty bind_address or * specifies all interfaces.

Notably, the -R flag is often used with -N to prevent any command from running and just establish the port forwarding (reverse tunnel) for future use.

3.3. Examples

To understand the idea of reverse tunnels, let’s see some examples.

First, we establish the basic template:

ssh -N -R [<remote_server_bind_address>]:<remote_server_bind_port>:<client_network_host>:<client_network_host_port> [<remote_server_user>@]<remote_server_address>

Now, let’s show a fairly simple case:

$ ssh -N -R localhost:8888:localhost:7777 uxer@xost

Here, any connection to the remote server xost on its localhost interface at port 8888 goes to the client that ran the command (second localhost) at port 7777.

Of course, the first localhost doesn’t need to be explicit, as it’s the default:

$ ssh -N -R 8888:localhost:7777 uxer@xost

This command should function in more or less the same way. Naturally, if we want to bind to another network interface on the remote server, we can do so via another address or a socket.

Lastly, when connecting to a client network machine (instead of locally to the client), we can specify the desired address and port in place of client_network_host:client_network_host_port:

$ ssh -N -R 8888:<client_network_host>:<client_network_host_port> uxer@xost

So, let’s do that:

$ ssh -N -R 8888:192.168.6.66:6666 uxer@xost

Thus, any connection to the loopback interface on xost at port 8888 goes to 192.168.6.66 at port 6666 in the client network.

4. Forwarding Port Issues

Different problems can arise during port forwarding and remote tunneling with SSH. Let’s explore some of them and see potential solutions.

4.1. Syntax Problems

It may take some time to understand the exact way each part of the -R flag value works. Considering this, we might end up mixing different addresses or ports, which can lead to unexpected errors and issues with accessibility.

For instance, if we use the remote_server_bind_address in the place of the client_network_host, we might see an error about an unknown host:

$ ssh -N -R 8888:xost:7777 uxer@xost
[...]
connect_to xost port 7777: failed

Furthermore, as with other networking tools, specifying exact parameters is paramount when constructing the SSH command for port forwarding. A missing or misplaced colon or bad ports and addresses can be hard to troubleshoot.

Another example involves the forced use of IPv6 addresses, which might need to be enclosed in square brackets.

To avoid such problems, checking the syntax and values multiple times is critical.

4.2. Privileged Ports

Reverse tunnels are a common occurrence within environments without root access.

To that end, attempting to use a port number below 1024 can result in permissions issues:

$ ssh -N -R 1000:xost:1100 uxer@xost
[...]
bind: Permission denied

The Permission denied error can be especially hard to diagnose when dealing with different machines, networks, passwords, and keys within the context of SSH. However, it usually just means that uxer is unable to bind port 1000 due to insufficient permissions.

Although the errors may be different, the remedy is the same: unless strictly necessary to do otherwise, it’s best to use port numbers above 1023.

4.3. Busy Ports

Commonly, when a forwarding fails, we see a fairly ambiguous error:

$ ssh -N -R *:8080:localhost:6666 uxer@xost
[...]
Warning: remote port forwarding failed for listen port 8080

Yet, one very common reason for issues when using reverse tunnels is a busy port.

In fact, checking whether a tunnel is up on the remote side might prove insightful even when there are no errors:

$ ss -tnlp | grep :8080
LISTEN 0      1            0.0.0.0:8080       0.0.0.0:*    users:(("nc",pid=10666,fd=3))

In this case, we see a netcat (nc) command uses the port we need for the tunnel. Thus, we can either stop that process or use a different port or IP version. However, the latter could happen unintentionally.

4.4. Wrong Internet Protocol (IP) Version

On top of the tunnel state, we can also see tunnel configurations by checking the remote server:

$ ss -tnlp | grep :8080
LISTEN 0      1            0.0.0.0:8080       0.0.0.0:*    users:(("nc",pid=10666,fd=3))                                 
LISTEN 0      128            [::1]:8080          [::]:*    users:(("sshd",pid=10667,fd=5))

In this case, both nc and sshd listen on 8080 after we’ve established a reverse tunnel without errors. However, sshd uses IP version 6 (IPv6), while nc works over IPv4 in this case. This might be desired behavior, but it could also be misleading when a successfully established tunnel is seemingly inaccessible through a given IP version.

4.5. Filtered Port

In specific cases, the remote_server_bind_port might end up being filtered. Let’s see what this may end up meaning for potential clients.

First, we establish a reverse tunnel:

$ ssh -N -R *:12345:localhost:6666 uxer@xost

Now, from another machine, we attempt to send a request to xost:12345:

$ curl xost:12345
curl: (7) Failed to connect to xost port 12345: Couldn't connect to server

At this point, we might doubt that the initial tunnel is working. However, it could be the case that port 12345 is inaccessible from outside the remote server xost. For example, we may need to configure a firewall or router. Yet, there are some ports that might even be filtered by the Internet Service Provider (ISP).

As already mentioned, reverse tunnels solve such filtering issues automatically on the client side for the client itself, but not beyond.

4.6. Remote SSH Server Configuration

One rare but possible reason for a reverse tunnel setup problem is an incorrect or unexpected configuration of the SSH server.

For instance, there are a couple of key settings for port forwarding:

• AllowTcpForwarding
• GatewayPorts

So, with the value of AllowTcpForwarding, we can enable (yes, default) or disable (no) TCP port forwarding, but we can also force only reverse and no local tunnels (remote) or only local instead of reverse tunnels (local) to be accepted.

On the other hand, GatewayPorts can be enabled (yes), disabled (no, default), or clientspecified. The latter enables the SSH client to specify the address. Since the default is no, ports usually only bind to 127.0.0.1 (loopback) on the remote host.

In case forwarding is disabled or a forbidden bind address is requested, we see the generic error:

$ ssh -N -R 0.0.0.0:8080:localhost:22 uxer@xost
[...]
Warning: remote port forwarding failed for listen port 8080

Still, if we know about these parameters, checking and correcting them in the sshd_config file (with a potential service restart) might solve some issues.

5. Summary

In this article, we talked about reverse SSH tunnels, i.e., SSH port forwarding, and issues that we might encounter when trying to use them.

In conclusion, although a reverse SSH tunnel is a fairly complex feature with many dependencies, by performing checks and diagnostics, we can deduce the correct reason for any failures and successfully set up stable forwarding.