Spotting cyber attackers with encrypted traffic analysis

Cybersecurity wisdom says that network traffic should be encrypted to protect against eavesdropping and man-in-the-middle attacks, typically using HTTPS (TLS), Secure Shell (SSH), and a range of different VPN protocols.

The growth rate of Internet encryption demonstrates this, with Fortinet reporting that “the total percentage of encrypted web traffic is now around 85%, up from just 55% in Q3 of 2017” and that it is a “larger and larger slice of a steadily increasing pie.”

But the popularity of encrypted network traffic poses a security challenge for any high-capacity network. Intrusion detection systems, firewalls, and other network security devices need to read the contents of packets to detect possible cyber threats, with data showing a 400% increase in TLS-based phishing threats from the previous year.

So, the question we need to ask is this: When encryption is being used, how can defenders spot malicious traffic?

ENCRYPTED TRAFFIC ANALYSIS

On a service provider’s network, there is no simple answer. Corporate networks can block ports and protocols in a way no service provider can. And yet the operators of networks passing large traffic flows know that malicious files hide within their traffic and that they must somehow locate them.

Effective cybersecurity becomes digital detective work, where teams use an overlapping series of techniques and technologies to sift traffic flows for clues pointing to anything unusual.  

The blunt force approach is to use TLS inspection to decrypt the traffic, examine its contents, before re-encrypting it and sending it on its way. This is a heavy-duty technique and increasingly unpopular and unsuited to network operators who must ensure security and compliance, as well as Quality of Service for the end users. For this reason, service providers should turn to subtler techniques. 

JA3 FINGERPRINTING

Whenever a data transmission session is established, the client and server conduct a handshake process to establish communication parameters. TLS sessions always start with a TLS handshake. It’s possible to identify specific elements within the TLS handshake, which result in creating a unique signature, or ‘fingerprint’ for each TLS handshake. Due to the unique nature of the fingerprint, this can be utilised and cross reference against a list of fingerprints enabling the detection of anomalous activity, including possible Indicators of Compromise (IoCs).

A popular method for this is JA3 fingerprinting
, a methodology for Encrypted Traffic Analysis (ETA). Although the content of TLS sessions is in ciphertext, the TLS handshake itself is in cleartext.

A significant amount of malicious encrypted traffic comes from the command and control servers that cyber attackers use to conduct activities such as distributed denial of service (DDoS), attacks through botnets, modular malware campaigns, and various other sorts of advanced persistent threats (APTs). Malware will often use custom parameters when communicating via TLS to their command and control server. Using JA3 fingerprinting, these malicious encrypted traffic sessions can be recognised and blacklisted, thus preventing further such attacks. 

ANOMALY DETECTION

To detect anomalous behaviour, a baseline is necessary to define what is normal on a network. Which internet services or TCP/IP ports are used by the network’s various client and server machines? How much bandwidth is usually used in the different parts of the networks and in the cloud? What are the usual upload and download speeds? Which points in the network typically communicate with which other points? These are all metrics that can be acquired without any data decryption. 

Security baselines are established with data that’s acquired over time in order to avoid designing a system that generates excessive false positives. For example, payroll and accounting servers may be more active once a month or once bi-weekly, a traffic flow which might appear anomalous. For this reason, the baseline should be based on months of activity and adjusted periodically.

An intrusion detection or prevention system may have sensors which detect activity based on signatures or policies. The same applies to many types of firewalls. These devices won’t work very effectively if the data running through them is ciphertext. Fortunately, there are specific ways to detect anomalies and other indications of malicious behaviour in encrypted data.

INFERRING MALICIOUS ACTIVITY

Telesoft’s 400Gbps FlowProbe network traffic monitoring product gives service providers visibility on advanced cyber threats, including those using encryption. FlowProbe is designed to detect malicious activity at a great scale, passing an enriched flow record to the Telesoft Data Analytics Capability (TDAC) for network forensics against real time data, as well as historical data. What matters is the identity of a session, not whether it is encrypted or not.

For more information on how Telesoft can address these issues, discover our Cyber Platform products.