How Do You Detect Malicious Packets If They're Encrypted?

Jul 11, 2017

Malware poses an increasing threat to network devices. Recently, malware has been used to install ransomware on computers, which encrypts data on the computers' hard drives and demands money for the data to be decrypted (essentially holding a computer’s data hostage and demanding a ransom). A logical response to help mitigate such threats is to have a network-based device analyze traffic flows for those offending packets.

However, what if the malware concealed itself to avoid detection? For example, some malware uses TLS-based encryption (where TLS stands for Transport Layer Security), thus preventing traditional network-based threat scanners from inspecting it. Interestingly, malware’s use of TLS-based encryption is growing rapidly. Consider that in August 2015, only 2.21 percent of malware attacks used TLS, while the percentage of malware attacks using TLS in May 2017 had grown to 21.44 percent.

This increasing threat begs the question, “How do we protected against malware if we can’t even read the packets containing the malware?” One solution some companies have used is to decrypt traffic for inspection. However, this can be very processor intensive (thus very expensive), and the decryption of traffic might violate some companies’ security policies.

So, what is a network engineer to do? Well, I just returned from Cisco Live US 2017 (CLUS17) in Las Vegas, and a very popular topic was a technology Cisco recently rolled out called Encrypted Traffic Analysis (ETA). As the name suggests, ETA can check encrypted traffic for offending payloads. At first, such a concept sounded counterintuitive to me, but then I was listening to a presentation that started to make sense of this concept. The specific analogy that resonated with me was this: When you go to a doctor for an illness. The doctor might not be able to immediately determine what’s wrong with you. However, they can do some tests (e.g. blood work, EKG, blood pressure, etc.), and the test results can indicate why you're feeling sick.

Similarly, ETA looks for symptoms of malware in encrypted traffic. Using huge data sets, Cisco has been able to determine (and is continually refining that determination) some characteristics of encrypted traffic carrying both benign and infected packets. For example:

  • The ClientHello Message: At the beginning of a TLS session, a client sends a ClientHello message. This initial packet is not encrypted, because it contains information used to negotiate the encryption. For example, the ClientHello message contains a listing of cipher suites that it supports, and which cipher suites it prefers to use. A cipher suite contains parameters used to negotiate the formation of an encrypted TLS connection. Examples of these parameters are: the type of authentication used, the type of encryption used, and the algorithms used to exchange keys. The list of cypher suites varies widely based on the type of client (e.g. desktop vs. mobile) and type of application. This means if suspicious cypher suite listings (which have been learned to accompany malicious traffic) are detected, the traffic can be blocked.
  • Packet Lengths and Times: Cisco again turns to its huge data set to learn the lengths of packets for specific connection types and time intervals between packets required to complete specific requests. If the packets making a request in an encrypted TLS connection have a statistically significant variation from known packet length and time values for benign traffic making similar requests, ETA can conclude the traffic probably contains malware.

We can use traditional network monitoring tools, such as NetFlow, to monitor encrypted traffic for suspicious characteristics. However, some of Cisco’s newest devices, such as the Cisco Catalyst 9300 Series switch, come with native Encrypted Traffic Analysis features. The results are impressive. Depending on the value used for the statistical threshold (referred to as an alpha value in statistics), ETA can currently achieve 99.99 percent accuracy in detecting benign packets and 85.80 percent accuracy in detecting malware packets.

Kevin Wallace, CCIEx2 (R/S and Collaboration) #7945