Determining the Correct Ports for IPSec/LT2P

1. Introduction

Configuring the correct ports for Internet Protocol Security (IPSec) and Layer 2 Tunneling Protocol (L2TP) is critical when setting up a secure virtual private network (VPN) connection. As system administrators, we often use these two protocols together to create encrypted tunnels that protect data as it travels across public networks. However, getting them to work properly requires an understanding of which ports we need to open on our firewall or router.

In this tutorial, we’ll discuss which ports are necessary for different IPSec and L2TP configurations, whether our server is behind Network Address Translation (NAT) or directly exposed to the internet. We’ll also examine which ports and protocols we need to open or forward to ensure seamless and secure VPN connectivity. Let’s get started!

2. Understanding IPSec and L2TP

Before we dive into port configuration, let’s start with a high-level overview of IPSec and L2TP, which will help contextualize why certain ports and protocols are necessary.

IPSec is a suite of protocols for encrypting and securing IP traffic. It’s common in VPN setups to ensure that data sent across public networks is protected from interception or tampering. IPSec offers features like authentication, data integrity, and encryption, making it a go-to solution for securing communications over the Internet.

On the other hand, L2TP is a tunneling protocol that doesn’t provide encryption on its own, but we often pair it with IPSec for security. L2TP creates the tunnel for our VPN, while IPSec adds the encryption layer, ensuring that the data within the tunnel remains secure.

However, when combined (often referred to as L2TP/IPSec), these protocols work together to create a secure VPN tunnel that allows encrypted communication between remote clients and servers. But to get these protocols to function correctly, we need to open certain network ports to allow the traffic to pass through our firewall or router.

3. Common Ports and Protocols for IPSec/L2TP

To ensure a proper connection, we need to understand the ports’ association with IPSec and L2TP. Each protocol operates on specific ports, and depending on our network setup, we might require additional ports or protocols.

3.1. Key Ports for IPSec

Let’s start by breaking down the common ports and protocols IPSec uses:

• UDP 500 (IKE – Internet Key Exchange): This is one of the most critical ports for IPSec. IKE negotiates and manages security keys for the IPSec connection. Without UDP port 500 open, IPSec will not be able to establish secure connections between the VPN client and server.
• UDP 4500 (NAT-T – NAT Traversal): NAT-T is an extension to IPSec that allows IPSec traffic to pass through devices performing NAT. In scenarios where either the VPN client or server is behind a NAT, IPSec encapsulates its packets within UDP 4500 to allow them to traverse the NAT device without issues.

These two ports are critical for IPSec to function properly.

3.2. Core IPSec Protocols

Now, let’s move on to the core protocols that IPSec uses for security. In addition to UDP ports, IPSec relies on certain core protocols to function:

• ESP (Protocol 50 – Encapsulating Security Payload): ESP is one of the core protocols IPSec uses to provide encryption, data integrity, and confidentiality for IP traffic. Unlike UDP or TCP, ESP is a protocol of its own and doesn’t use a port number, but we still need to allow it through the firewall. It handles the actual encryption of the data we’re transferring over the VPN.
• AH (Protocol 51 – Authentication Header): AH provides integrity and authentication for IPSec packets, ensuring that data is not tampered with during transmission. While not always necessary (as ESP can provide both encryption and authentication), we can sometimes use AH in certain IPSec configurations. Like ESP, it’s a protocol and doesn’t rely on a port number.

Both ESP and AH ensure that IPSec traffic remains secure.

3.3. Key Port for L2TP

Next, we need to address the specific port L2TP uses:

• UDP 1701 (L2TP): L2TP uses this port to establish the VPN tunnel. However, we should note that L2TP doesn’t provide encryption on its own, and this is why it’s often paired with IPSec. For security, we should only allow traffic on UDP 1701 after securing the connection with IPSec.

In short, these are the key ports and protocols that we need to manage to ensure proper IPSec and L2TP functionality. The specific ports and protocols required will depend on our network setup, which we’ll explore in the following sections.

4. IPSec/L2TP Without NAT

When working with a firewall or router in a routed environment without NAT, configuring ports for IPSec and L2TP is more straightforward. In this scenario, we don’t need to deal with the complications NAT introduces, such as NAT-T. This simplifies the setup since we only need to ensure that the correct ports and protocols are open to allow traffic.

4.1. Essential Ports and Protocols in Non-NAT Environments

Let’s see a breakdown of the essential ports and protocols to open in a non-NAT environment:

• UDP 500: We need to open this port to allow IKE traffic to pass between the client and the VPN server. IKE handles the negotiation of the security association (SA), which establishes the encryption and authentication methods IPSec uses. Without this port, we cannot establish IPSec connections.
• UDP 4500: While NAT environments mainly require UDP 4500, we can open this port regardless, as some modern firewalls and VPN implementations might still leverage it, even in non-NAT setups. Although its primary usage is for NAT traversal, it doesn’t hurt to have it open as a precaution.

These ports are essential to allow proper VPN traffic in non-NAT setups.

4.2. Managing IPSec Protocols in Non-NAT Setups

In addition to opening the ports, we need to ensure our firewall allows the relevant IPSec protocols:

• ESP: This protocol handles the actual encryption and decryption of the data packets. Since ESP operates on its protocol (rather than a specific port), we should ensure that our firewall allows Protocol 50 to pass through without blocking it.
• AH: Although AH is less common (since ESP can handle both encryption and authentication), some setups may still require Protocol 51 for integrity and authentication checks. If our VPN setup includes AH, we’ll need to ensure that our firewall’s configuration allows Protocol 51.

Ensuring these protocols are open is vital for IPSec functionality.

4.3. Handling L2TP Traffic Securely

Finally, we need to secure L2TP traffic by managing this specific port:

• UDP 1701: L2TP uses the UDP 1701 port to establish the VPN tunnel. However, in a proper L2TP/IPSec setup, we should only allow traffic on UDP 1701 after securing the connection with IPSec. This means we should ensure that the firewall allows inbound traffic on UDP 1701 only if it’s coming through an IPSec-protected channel.

In this basic, non-NAT scenario, our primary concern is opening these essential ports and protocols to ensure IPSec and L2TP work smoothly. Since we don’t have to deal with NAT traversal, the configuration is simpler, and the likelihood of encountering issues is lower.

5. IPSec/L2TP Behind NAT: Port Forwarding

When we place our IPSec/L2TP VPN server behind a NAT device, things get a bit more complex. NAT devices modify the source or destination IP addresses of packets as they pass through, which can interfere with protocols like IPSec that depend on end-to-end packet integrity.

Therefore, to make IPSec/L2TP work properly in NAT environments, we need to handle NAT traversal and port forwarding carefully. Let’s break down what we need to do in such scenarios.

5.1. Why NAT Affects IPSec Traffic

The core issue with NAT and IPSec lies in how IPSec secures packets. IPSec, particularly the ESP protocol, encrypts the entire packet, which includes the source and destination IP addresses. When a NAT device changes these addresses, it breaks the IPSec integrity checks, causing the connection to fail.

However, NAT-T came around to resolve this issue. By encapsulating IPSec traffic in UDP packets, NAT-T allows the traffic to pass through NAT devices without altering the critical parts of the encrypted packet.

5.2. Ports and Protocols to Forward in a NAT Environment

If our VPN server sits behind a NAT device, we’ll need to set up port forwarding to ensure that the traffic from the VPN clients can reach the server.

Let’s see the key ports and protocols we need to forward:

• UDP 500 (IKE): Just like in non-NAT environments, we need to forward UDP port 500 to the VPN server. This is the port IKE uses to negotiate security keys for the IPSec connection.
• UDP 4500 (NAT-T): This port is crucial for NAT environments. When IPSec traffic needs to traverse NAT, it gets encapsulated in UDP packets using port 4500. Forwarding this port allows the encapsulated traffic to pass through the NAT device without breaking the IPSec security associations.
• UDP 1701 (L2TP): Forwarding UDP 1701 is a requirement for the L2TP protocol, but as always, we should only allow it after IPSec has secured the connection. This means that we should only accept L2TP traffic on this port if it’s coming through an IPSec-protected tunnel.

However, a question arises here: Should we forward ESP (Protocol 50) and AH (Protocol 51)?

When our server is behind NAT, there’s no need to forward the ESP or AH protocols. This is because NAT-T encapsulates all IPSec traffic within UDP, using port 4500. This means the ESP and AH protocols never need to traverse the NAT device directly – they’re encapsulated inside the UDP packets.

5.3. Example iptables Configuration for NAT

On our Linux server, we need to configure iptables for a NAT environment.

Let’s see how to forward the necessary ports:

$ iptables -A PREROUTING -t nat -p udp --dport 500 -j DNAT --to-destination <VPN_SERVER_IP>:500
$ iptables -A PREROUTING -t nat -p udp --dport 4500 -j DNAT --to-destination <VPN_SERVER_IP>:4500
$ iptables -A PREROUTING -t nat -p udp --dport 1701 -j DNAT --to-destination <VPN_SERVER_IP>:1701

We should replace <VPN_SERVER_IP> with the internal IP address of our VPN server.

6. Using iptables to Configure Ports for IPSec/L2TP

Let’s discuss how to configure a firewall using iptables to allow IPSec/L2TP traffic through the correct ports. Whether our VPN server is directly exposed to the internet or behind NAT, iptables is a powerful tool to manage firewall rules.

6.1. iptables Rules for Non-NAT Environments

If our VPN server maintains a direct connection to the internet (that is, not behind NAT), we’ll need to open the following ports and protocols in our firewall:

$ iptables -A INPUT -i eth0 -p udp --dport 500 -j ACCEPT
$ iptables -A INPUT -i eth0 -p udp --dport 4500 -j ACCEPT
$ iptables -A INPUT -i eth0 -p 50 -j ACCEPT   # ESP protocol
$ iptables -A INPUT -i eth0 -p 51 -j ACCEPT   # AH protocol
$ iptables -A INPUT -i eth0 -p udp -m policy --dir in --pol ipsec -m udp --dport 1701 -j ACCEPT

Here, the first rule opens UDP 500, which is necessary for IKE. The second rule opens UDP 4500, which is required for NAT-T (and recommended even in non-NAT setups). Then, the last rule allows UDP 1701 traffic for L2TP, but only when it’s part of an IPSec-secured connection. The -m policy and –pol ipsec flags ensure that only traffic protected by IPSec can use this port, adding an extra layer of security.

6.2. iptables Rules for Servers Behind NAT

If our server is behind a NAT device, we’ll need to forward the appropriate ports, as discussed earlier, and apply similar rules to the server’s internal firewall.

Let’s see an example iptables configuration for a server behind NAT:

$ iptables -A INPUT -i eth0 -p udp --dport 500 -j ACCEPT
$ iptables -A INPUT -i eth0 -p udp --dport 4500 -j ACCEPT
$ iptables -A INPUT -i eth0 -p udp -m policy --dir in --pol ipsec -m udp --dport 1701 -j ACCEPT

In this case, we didn’t include ESP and AH because NAT-T encapsulates IPSec traffic in UDP packets over port 4500.

7. When ESP and AH Protocols Are Necessary

In most IPSec/L2TP VPN configurations for remote access (such as using a client to connect to a server), we generally don’t need to worry about the ESP and AH protocols directly, especially when using NAT-T. However, when setting up site-to-site VPNs, these protocols become more critical.

7.1. Understanding Site-to-Site VPNs

A site-to-site VPN connects two separate networks, typically over the internet, and allows traffic to pass securely between them as if they were on the same local network. For instance, two branch offices might use a site-to-site VPN to communicate over the public internet securely.

However, unlike client-to-server VPNs, which primarily need UDP 500 and 4500, site-to-site VPNs use the ESP and AH protocols more explicitly because both endpoints are likely to be directly connected to the internet without NAT. These protocols enable the exchange of encrypted traffic between routers or firewalls at both ends.

7.2. When We Need to Enable ESP and AH

In a site-to-site VPN setup, we need to allow the ESP and AH protocols through the firewall to enable encrypted VPN traffic to flow between both networks. ESP handles encryption and encapsulation of data within the VPN tunnel. It’s the core protocol that provides confidentiality by encrypting the data packets.

Then, AH provides data integrity and authentication, ensuring that the data hasn’t been tampered with. It can also provide replay protection. While AH isn’t always necessary (since ESP can handle both encryption and authentication in most setups), some VPN configurations might use AH in combination with ESP for additional security.

7.3. Firewall Considerations for Site-to-Site VPNs

If our IPSec VPN is running in site-to-site mode, we need to configure our firewall to allow ESP and AH protocols on both ends:

$ iptables -A INPUT -i eth0 -p 50 -j ACCEPT   # Allow ESP (Protocol 50)
$ iptables -A INPUT -i eth0 -p 51 -j ACCEPT   # Allow AH (Protocol 51)

In addition to these, we’ll still need to open UDP 500 for IKE and UDP 4500 for NAT-T if the site-to-site VPN passes through NAT on one or both sides.

7.4. Site-to-Site NAT Considerations

In some cases, one of the VPN endpoints might be behind a NAT device. If this is the case, NAT-T will encapsulate ESP and AH traffic inside UDP 4500 packets, so we won’t need to forward Protocols 50 and 51. Instead, we should ensure we forward UDP 500 and UDP 4500.

7.5. iptables Rules for Site-to-Site VPNs

For a site-to-site VPN setup without NAT, here’s a basic iptables ruleset:

$ iptables -A INPUT -i eth0 -p udp --dport 500 -j ACCEPT  # IKE
$ iptables -A INPUT -i eth0 -p 50 -j ACCEPT               # ESP
$ iptables -A INPUT -i eth0 -p 51 -j ACCEPT               # AH

However, if it involves NAT, we should adjust the rules to include UDP 4500 for NAT-T:

$ iptables -A INPUT -i eth0 -p udp --dport 500 -j ACCEPT  # IKE
$ iptables -A INPUT -i eth0 -p udp --dport 4500 -j ACCEPT # NAT-T

In short, the ESP and AH protocols are necessary for site-to-site VPN configurations where there is traffic encryption directly between routers or firewalls. We may not need these protocols for typical remote-access VPNs behind NAT, but they are critical for direct internet-connected site-to-site VPNs.

8. Conclusion

Configuring the correct ports for IPSec/L2TP is essential to ensuring a functional and secure VPN setup. In this article, we’ve covered the key ports and protocols needed for both non-NAT and NAT environments. We also discussed the specific configurations required for site-to-site VPNs and highlighted best practices to secure our setup.