Capture the SSL Handshake with tcpdump

1. Overview

SSL is the most common protocol for exchanging encrypted data over a TCP connection. And in order to establish an SSL connection, the two endpoints must exchange public keys, encryption algorithm, protocol version, and so on. This exchange is known as an SSL handshake.

Since this is an asymmetric key-certificate exchange, there’s often a chance that the handshake will fail. For example, if the server has an expired certificate or if the client and server can’t negotiate the SSL/TLS protocol version, the handshake will fail.

In most cases, we can find the failure reason by analyzing the SSL handshake messages between the client and server. In this tutorial, we’ll study a way to capture these messages over the network.

2. The tcpdump Command

The tcpdump command allows us to capture the TCP packets on any network interface in a Linux system.

Generally, a lot of TCP traffic flows in a typical SSL exchange. Although tcpdump is quite useful and can capture any amount of data, this usually results in large dump files, sometimes in the order of gigabytes. Such dump files are sometimes impossible to analyze. For example, it would require a lot of resources in analyzing such dumps in Wireshark.

To overcome this problem, the tcpdump command provides some filtering options. Thus, only those TCP packets that satisfy the filtering conditions are captured in the output dump.

3. Introduction to the SSL Handshake

Let’s quickly go through the messages that the client and server exchange during the SSL handshake:

1. Client Hello – Originated by the client. It contains the protocol version, cipher suites supported by the client, and a secured random number.
2. Server Hello – Returned by the server in response to the Client Hello. Contains the protocol version chosen by the server, selected cipher suite from the client’s list, encryption algorithm, and other TLS version-specific extensions.
3. Server Certificate – Originated by the server. Contains the public certificate chain that the client will authenticate.
4. Certificate Request – Originated by the server. This message is only sent if the server also needs to authenticate the client, as is the case in two-way SSL.
5. Server Hello Done – Originated by the server. Indicates the end of Server Hello.
6. Client Certificate – Returned by the client in response to Client Request. Client sends its certificate chain to the server.
7. Client Key Exchange – Originated by the client. It generates a pre-master secret and encrypts it with the server’s public certificate. It then sends the master secret to exchange the encryption algorithm with the server.
8. Certificate Verify – Originated by the server. This indicates successful authentication of the client’s certificate.
9. Finished – Sent by both the client and the server to indicate successful authentication and key exchange.

If there are some SSL failures during connection establishment, analyzing the above messages is a good starting point.

Although we’ll not discuss these messages in detail, it’s important to realize that these messages are part of the TCP data packets. Therefore, if we want to capture only these messages, we need advanced filtering options compared to the ones we studied in the last section.

With this in mind, let’s explore some of the data filtering options in tcpdump and see how we can use them to filter only SSL handshake messages.

4. Filtering SSL Handshake Messages in tcpdump

In addition to the metadata like port or host, the tcpdump command also supports filtering on the TCP data. In other words, tcpdump allows us to match the data bytes in the packet with a filter expression. For example, we can filter packets with certain TCP flags:

tcpdump 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0'

This command will capture only the SYN and FIN packets and may help in analyzing the lifecycle of a TCP connection.

In the same way, we can filter SSL handshake messages if we know the structure of data bytes. From the TLS specification, we know that every message in the handshake protocol starts with a unique numerical value. For example, all handshake message contains 22, represented as 0x16 in hex, as the first data byte:

So, based on this fact, let’s see how we can filter the handshake messages.

4.1. Capturing Client Hello

Suppose we want to analyze SSL connection establishment attempts from a client. For this, we must check the Client Hello message between the client and the server. The Client Hello messages contain 01 in the sixth data byte of the TCP packet. Thus, to filter such packets:

tcpdump "tcp port 8081 and (tcp[((tcp[12] & 0xf0) >>2)] = 0x16) \\
  && (tcp[((tcp[12] & 0xf0) >>2)+5] = 0x01)" -w client-hello.pcap

Let’s understand the different parts of the command options:

• tcp port 8081 – captures packets only on port 8081, assuming this is the SSL port of the application server
• and (tcp[12] & 0xf0) – reads the 13th byte of the packet and keeps the higher 4 bits
• && ((tcp[12] & 0xf0) >>2) – when we multiply the above by 4, it gives the TCP header size

As we can see from the SSL dump above, the TLS header precedes the TCP data packet. So, to get the first and sixth data byte, we need to calculate the TCP header size and skip matching these bytes. The second and third terms above do just that.

Now, tcp[TCP header size] points to the first byte of the data in the packet. And thus the term, tcp[((tcp[12] & 0xf0) >>2)] = 0x16 checks whether this byte is equal to 22, the numerical code for SSL handshake. And, tcp[((tcp[12] & 0xf0) >>2)+5] = 0x01 will filter packets where the sixth byte is 1, representing Client Hello.

Similarly, we can capture any handshake message we discussed earlier. For example, we can use tcp[((tcp[12] & 0xf0) >>2)+5] = 0x02 for Server Hello messages.

4.2. Capturing Packets with Specific TLS Version

The SSL protocol has been evolving over time. After SSLv3, the protocol was succeeded by TLS, which is mostly similar. Modern applications generally exchange messages over TLSv1.3. However, many still support TLSv1.0, TLSv1.1, and TLSv1.2 for backward compatibility.

The TLS specification assigns a unique numerical code to every TLS version:

• SSLv3 – 0x300
• TLSv1.0 – 0x0301
• TLSv1.1 – 0x0302
• TLSv1.2 – 0x0303
• TLSv1.3 – 0x0304

In the SSL handshake message, the tenth and eleventh bytes of the data contain the TLS version. Therefore, a tcpdump filter can be applied:

tcpdump "tcp port 8081 and (tcp[((tcp[12] & 0xf0) >>2)] = 0x16) \\
  && (tcp[((tcp[12] & 0xf0) >>2)+9] = 0x03) \\
  && (tcp[((tcp[12] & 0xf0) >>2)+10] = 0x03)"

The terms tcp[((tcp[12] & 0xf0) >>2)+9] = 0x03 and tcp[((tcp[12] & 0xf0) >>2)+10] = 0x03 check the tenth and eleventh bytes to filter all packets over TLSv1.2. This command will capture all SSL handshake packets where TLSv1.2 is exchanged.

4.3. Capturing Application Data Packets Over TLS

So far, we have only captured SSL handshake messages. Once the handshake is finished, the client and server can exchange the application data. Such application data packets also contain the TLS version in the second and third data bytes:

tcpdump "tcp port 8081 and (tcp[((tcp[12] & 0xf0) >>2)] = 0x17) \\
  && (tcp[((tcp[12] & 0xf0) >>2)+1] = 0x03) \\
  && (tcp[((tcp[12] & 0xf0) >>2)+2] = 0x03)" -w appdata.pcap

Here, we’ve added a filter to capture packets that have 17 in the first byte and 03 in the second and third bytes. This is because 17 is the numeric code for Application Data packets, and 0303 denotes TLSv1.2, as we have seen before.

4.4. Capturing SSL Connection Failures

To filter failures, we’ll check the first byte, which contains 15 or 21, based on the failure:

tcpdump "tcp port 8081 and (tcp[((tcp[12] & 0xf0) >>2)] = 0x15) || (tcp[((tcp[12] & 0xf0) >>2)] = 0x21)" -w error.pcap

This command will capture packets where the first data byte is either 15 or 21.

5. Conclusion

In this article, we discussed tcpdump filters to match the TCP data in a packet with an expression. Using this knowledge, we can easily capture packets where data matches the filter expression.

We later used this approach to capture the SSL handshake packets by matching a unique numeric code for each message.