Timeout Issue with Port Forwarding in Kubernetes

1. Introduction

Port forwarding is a simple way to access an application on a pod, even if it’s not exposed externally. However, forwarding a local port to a port on a pod can cause a timeout issue.

In this tutorial, we’ll learn three different methods to address the timeout issue with port forwarding in Kubernetes.

First, we’ll understand what port forwarding is in the context of Kubernetes and what it’s used for. Next, let’s take a look at the timeout issue that can occur with port forwarding in Kubernetes. Finally, we’ll examine the three methods for addressing the issue.

2. What’s Port Forwarding in the Context of Kubernetes?

In Kubernetes, port forwarding allows us to easily access applications running inside a pod, even if we don’t expose them externally. Therefore, Kubernetes users widely use this feature for testing, debugging, and troubleshooting purposes.

In practice, port forwarding can redirect all traffic from a local port to another port on a Kubernetes pod. Thus, we can map a local port to the same one on a pod or map different ports.

However, we aren’t directly communicating with the pod when using the port forwarding mechanism in Kubernetes. Kubernetes establishes a connection to the pod and another to our local machine. Once these connections are established, Kubernetes is a proxy between the local machine and the specific pod.

3. Timeout Issue Example

Kubernetes provides the port forwarding mechanism via the kubectl command line tool. Thus, by using the command below, we can forward a local port to a port on a specific pod:

$ kubectl port-forward <pod-name> <localhost-port>:<pod-port>

For example, let’s say we have a pod called my-server with a web server listening on port 80. In this case, we want to forward all traffic going to localhost:8088 to the my-server pod on port 80. We can do this using the following command:

$ kubectl port-forward my-web-server 8088:80
Forwarding from 127.0.0.1:8088 -> 80
Forwarding from [::1]:8088 -> 80

When we access http://localhost:8088, we’ll get a successful response from the server. In addition, in the log of the kubectl port-forward command, we’ll see the following message:

Handling connection for 8088

However, after the connection has been idle for a long time, communication with the pod via local port 8088 may not work. In this case, the log of the kubectl port-forward command will show a message that looks like this:

E1125 10:46:28.623549	17437 portforward.go:235] lost connection to pod

The problem arises because Kubernetes defines the maximum time a streaming connection can be idle before automatically closing the connection. Therefore, when this timeout is exceeded, port forwarding no longer works.

Next, we’ll look at three different methods of addressing this issue.

4. Set a Longer Timeout for Idle Connections

The streamingConnectionIdleTimeout parameter defines the maximum time a streaming connection can be idle. By default, this parameter is set to 4 hours. This means that Kubernetes will automatically terminate connections that have been idle for more than 4 hours.

At first, we could disable this parameter by setting the timeout to 0. However, by doing this, we’d be leaving the environment vulnerable to denial-of-service attacks, inactive connections, and running out of ephemeral ports. For this reason, we don’t recommend setting the timeout to 0.

A more appropriate solution is to set a longer timeout according to our needs. For this purpose, we can edit the Kubelet configuration file, located in /var/lib/kubelet/config.yaml. Here, all we need to do is set a new value for the streamingConnectionIdleTimeout parameter.

For example, we can set the timeout to 5 hours as follows:

streamingConnectionIdleTimeout: 5h0m0s

After modifying the Kubelet configuration file, we need to restart the Kubelet service:

$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet.service

Afterward, Kubernetes will allow connections to remain idle for up to 5 hours before terminating them (1 hour longer than the default in this case). Please note that this modification will affect all pods. Please note that this modification will affect all pods.

5. Recreate Port Forwarding Periodically

Instead of modifying the timeout for idle connections, we can prevent it from being exceeded. So, one possible approach is to periodically recreate the port forwarding connection whenever we’re close to exceeding the timeout.

At first, we might think that we just need to create a simple loop to execute the command every time. However, we should note that the command prompt gets stuck while executing the kubectl port-forward command. In other words, we can only visualize the connection logs and need to type Ctrl+C when we want to terminate the connection.

Therefore, recreating the port forwarding connection periodically requires an elaborate approach. The following shell script performs the necessary steps for this task:

#!/bin/bash

# Define the port forwarding command
command_to_run="kubectl port-forward <pod-name> <localhost-port>:<pod-port>"
time_to_wait="<time-in-seconds>"

# Function to run the command in the background
run_command() {
    # Run the command in the background
    $command_to_run &
    # Store the process ID of the command
    command_pid=$!
    echo "Port Forward command started with PID: $command_pid"
    # Wait for "time_to_wait" seconds
    sleep $time_to_wait
    # Kill the command process
    kill $command_pid
}

# Infinite loop to run the command periodically
while true; do
    run_command
done

In short, the script above creates the port forwarding connection, waits a certain amount of time, closes the connection, and then starts all over again. Note that we need to replace the parameters of the kubectl command, as well as the <time-in-seconds> parameter, according to our needs.

In addition, it’s important to optimize the script to recreate port forwarding as few times as possible. To do this, we just need to use a value that is very close to the timeout.

For example, assuming the timeout is 4 hours (the default value), we can recreate the port forwarding every 03:59:59 (3 hours, 59 minutes, and 59 seconds) by setting <time-in-seconds> = 14399.

6. Create a Loop to Keep the Connection Busy

Another way to address the timeout issue with port forwarding is to avoid leaving the connection idle for too long. This means that we should use the connection periodically.

A simple way to do this is to create a loop that periodically sends a request to the application running on the pod via port forwarding. A basic script might look like this:

$ while true ; do <request-command> ; sleep <time-in-seconds> ; done

In this case, we need to replace <request-command> according to the application running on the pod. For example, suppose that the pod is running a web server and the local port mapped by port forwarding is 8088. Then, we can replace the <request-command> in the script with a curl command, as follows:

$ curl http://localhost:8088

Similarly, depending on the application running on the pod, we can execute a database query, send a ping, or trigger any other type of stimulus.

This method is straightforward and has two other advantages. First, it doesn’t require any configuration on the Kubernetes cluster (such as changing the timeout, for example). Second, it doesn’t interrupt port forwarding for even one second (which could happen if we recreate the connection).

7. Conclusion

In this article, we’ve studied three different ways to solve the timeout issue with port forwarding in Kubernetes.

As we’ve learned, we can set a longer timeout, recreate port forwarding periodically, or even keep the connection busy by sending some requests.

Full Archive

About Baeldung