1. Introduction
Transmission Control Protocol (TCP) is responsible for transmitting a file or a message over a connected network. It uses flags to indicate a connection’s state and provide information for troubleshooting. In particular, the reset flag (RST) is set whenever a TCP packet doesn’t comply with the protocol’s criteria for a connection.
In this tutorial, we’ll go over the most common causes of the RST flag. But first, we’ll talk about the TCP flags in general.
2. TCP Flags
TCP flags are binary bits in the TCP header:
They convey the connection’s state and are helpful for troubleshooting. Each of them plays an important role in TCP communication such as initiating and closing connections or carrying data. Here, we’ll discuss six important control flags.
The SYN flag (synchronize) is used to initiate a connection. We set the ACK flag to acknowledge the safe receipt of the data packets, and the initiation/ tear down requests. The RST flag signifies the no-connection state or a non-compliance state of a request. The FIN flag denotes that the connection broke. Both the source and target send the FIN packets for the graceful connection termination.
We set the PSH flag if the receiver can pass the sent data on directly to the application instead of buffering. The urgent pointer (URG) flag means that a packet is a priority. In general, the receiver handles packets using queues. But, a packet with an active URG flag escapes the queue.
3. Causes of Sending RST Flag
The TCP RST flag signifies that we should immediately terminate the connection. It can happen because of many things, and here, we’ll talk about the most common causes. Later, we’ll go over two examples.
3.1. Non-existent TCP Port
Mostly, we’ll get an error “connection refused” if the client initiates a connection to a non-existing address. The error tells us that the target IP address of the TCP connection isn’t listening. That may happen if the target port doesn’t exist or the application isn’t running on the target server.
3.2. Aborting a Connection
Let’s say we are aborting a running application. This will trigger an interrupt to the process of communication and send an RST flag to a remote server with an error “Connection closed by peer”.
3.3. Asynchronous Connection
Let’s say the client-side is active but the sever side is inactive without the client’s knowledge. This may be due to an interruption at the physical layer that erases data in the transmission control block (TCB). Let’s suppose that the server restarts. Since the history of the earlier connection is missing, it sends an RST in response to the client’s request.
3.4. The Listening Endpoint Queue Is Full
We are aware that the listening endpoint has some queue. Let’s say the queues are not getting freed up quicker than the arriving connection requests, in such a condition RST is sent to the new connections.
3.5. Time-Wait Assassination
Let’s say the server received an ACK in response to a FIN request that it sent to the client and that the client is now in the Time-Wait state. In some cases, due to latency, the client receives old SEQ and ACK flags from the server. In response, the client sends ACK to the server. Since the server has no information about the already closed connection, it sends RST to the client.
3.6. Restricted Local IP Address
Let’s say we specified the listening endpoint as an IP address as 162.254.206.227:2106 instead of any local address such as *:2106 for security reasons.
If a client sends a request to any IP address other than 162.254.206.227:2106, the server will send RST back to the client.
3.7. Firewall in Transit
Sometimes, the firewall is configured to send RST due to a mismatch in credentials from the client or server. If TCP SYN crossing the firewall matches with an existing firewall session, the firewall will raise the RST flag. If the firewall session expires without the endpoint’s knowledge, then the endpoint sends RST.
3.8. TCP Buffer Overflow
The TCP acceleration on slow WAN links paired with fast LAN connections can trigger the flag. In such cases, the LAN populates the buffer faster than the endpoint drains it over the slower WAN.
4. RESET Sequence
To explain the RESET sequence, we’ll first go over two important stages of a TCP connection: establishing and closing it.
4.1. Establishing and Closing a Connection
A 3-way Handshake is achieved with SYN, SYN+ACK, and ACK packets to establish a new TCP connection:
We’ll be able to terminate an established connection gracefully by a 3-way handshake when a client sends a FIN and the server responds with a FIN+ACK. The client reciprocates with an active ACK flag. Therefore, a normal closing needs a FIN – ACK pair from each endpoint of a TCP connection:
4.2. Closing the Connection With the RST Flag
However, a connection may be abruptly closed by the server or client by sending an RST flag to the other.
Let’s consider the client sending an RST to the server. The client doesn’t expect an ACK in response from the server, nor does the server need to send a FIN/ACK exchange to terminate the connection:
The result of triggering the RST flag from one end leads to a request to other ends to stop communication all of a sudden. In the process, the packets in transit get discarded. The receiving end of the RST flag has to stop all communication with the other end.
5. TCP Reset Attack
Let’s now talk about a type of attack in which hackers send malicious TCP packets with the active RST flag to the server. These attacks disrupt the operation of a server leading to a Distributed Denial-of-Service (DDoS).
A TCP reset attack can terminate a TCP conversation when a server receives a TCP segment with the active RST. For example:
- The hacker sends SYN requests to a web hosting server,
- The server sends SYN-ACKs in response to each SYN request and waits for the three-way handshake.
- The hacker could send a spoofed packet with a TCP RST to one or both endpoints of the TCP connection causing the abrupt shutdown of communication.
Visually:
This makes the server immediately stop the connection.
6. Conclusion
In this article, we talked about the RST flag in the TCP packets. There are multiple scenarios where the RST flag is sent from one end to the other, causing abrupt termination of conversation, a half-open TCP connection, and a non-existent server.
Hackers with malicious intent often use the TCP RST flag to create havoc such as shutting down a website and denying genuine users access to a website. To ensure the continuity and safety of the TCP connection, we should constantly inspect the upcoming TCP flags.