Modern Cyber Defense — Part 6 — Threat Hunting
This article is in continuation of my series. While this article can be read independently, reading the earlier articles will provide the context of where I am coming from. You may read the earlier articles at Part 1, Part 2, Part 3, Part 4, or Part 5.
This article is about Threat Hunting. But, before we get there, it will be interesting to explore a few things. What is Threat Hunting? and where does it fit in to the overall cyber defense program?
For over two decades, Security Information & Event Management Systems (SIEM) have provided the detection layer in Security Operation Centers (SOC). The primary motivation has been compliance, but as attack behaviors evolved, the classic Log Management based SIEMs have given way to Analytics platforms that can do more. But think about what is the primary purpose for enterprises to invest in SOC? It is to discover potential incidents from raw logs, that could pose a threat to the enterprise.
It started off as a simple signature based detections, but has evolved into multiple ways to convert raw logs/ data into alerts, as can be seen below.
The above diagram shows my poor attempt at representing a high level SOC process workflow, where starting from left, raw logs/ data are fed into various kinds of correlation engines to to generate alerts, which are then triaged and analyzed to filter out potential incidents. Let us examine the three correlation engines in slight detail.
Signature Based: These are primarily compliance based and works extremely well when we know what to look for. For example, let us consider Brute Force. A typical signature based logic, found in almost all SIEM, is to detect X number of failed logins within Y minutes. But how do we determine X and Y? X could be 5 for one enterprise, or could be 20 for another. What if the attacker was successful in 3 attempts? What if the attack was low and slow, spread out beyond Y? So many variables here for such a simple rule. And that is why signature based rules are no longer adequate to provide complete threat coverage. They are still required though and can be the most resource efficient way to detect known bad.
Behavior Based: When attack complexity increases, signature based rules are no longer adequate. The next best thing is to detect based on known attacker behavior. For example, many attackers are known to favor credential dumping using tools like mimikatz. Can we build a detection logic to look for credential dumping? Can we look for Command & Control (C2) traffic in outbound network connection? Can we analyze outbound traffic to newly created domains or rare domains? Each of these are not specific signature based but based on typical attacker behavior. MITRE ATT&CK framework has provided us with an exhaustive listing of attack behaviors.
Anomaly Based: The third category is based on change of behavior. Discovering something that is not normal. For example, an odd hour authentication action by a user, or an abnormally large volume of data sent out from an endpoint. The challenge here is knowing what is normal, and then looking for indicators of abnormal activities. Using classical methods, it is not feasible to define normal behavior for thousands of users and/ or endpoints in an enterprise. We need data analytics and statistical analysis techniques to create data patterns and then look for anomalies.
A good SOC needs all three types of correlation to provide maximum threat detection coverage.
But what does this have to do with threat hunting? Everything, I say.
“because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns — the ones we don’t know we don’t know.”
— Donald Rumsfield
Although out of context, I think Donald Rumsfield got it absolutely right in his famous 2002 briefing. Signature based detections are good for known knowns. We know what the attacker does and how do they do it. We have signatures available for them and can create indicator based rules to detect them.
The known unknows are interesting. We know what the attacker does, but not how they do it. This is prime for behavior based detections, but these are hard and also many of the techniques do not lend themselves well for real time detection; and this is where Threat Hunting comes in. Threat hunting is the art of finding the unknowns in the environment, going beyond the traditional. We shall explore more on this later.
Lastly, the unknown unknowns. With attackers wielding a complex set of arsenal and advanced techniques, defenders need the power of modern data analytics and Machine Learning to crunch through all the raw data, derive normal behavior pattern and then look for deviations from that; all the time dynamically adjusting the normal pattern. I will try to explore more on this topic in another article.
In the above chart, we revisit the basic SOC process and try to link the types of correlation logic with detection techniques. We are still trying to generate alerts from raw logs/ data but identify the different ways we achieve the goal. As we see above, Threat Hunting primarily deals with finding out Known Unknows but can also be used sometimes to detect Unknown Unknowns.
A good SOC will have elements of all three detection types and not treat Threat Hunting as something abstract to be slapped onto SIEM or conducted using EDR/XDR. So, what is Threat Hunting and how do we do it?
NIST Risk Management Framework (RMF SP 800–53 Rev 5.1 and SP 800–53 B) describes Threat Hunting as:
Threat hunting is an active means of cyber defense in contrast to traditional protection measures, such as firewalls, intrusion detection and prevention systems, quarantining malicious code in sandboxes, and Security Information and Event Management technologies and systems. Cyber threat hunting involves proactively searching organizational systems, networks, and infrastructure for advanced threats.
All emphasis in the above quote is mine.
There are quite a few strategies to conduct Threat Hunting; Structured or Unstructured, and a few models.
Structured Hunt squarely lies in the ‘Known Unknown’ category. We know something about the potential attacker, either Indicators of Compromise (IoC) or their Tactics, Techniques and Procedures (TTPs) and the threat hunter actively looks for signs of a potential compromise using these.
Unstructured hunt is typically based on a trigger. It could be triggered by a Threat Intelligence input or a hypothesis. Hunter does not have a process or goal in mind but decides on next course of action simply based on findings in the previous step(s).
Threat Hunt Models
Threat Intelligence based hunting: Hunt is triggered by a Threat Intel input where IoCs are known. Hunter runs a search for the IoC in historical data sets to determine if the enterprise has evidence of those IoC. This is probably the most widely used model of threat hunting today, but in a not so good way.
Remember the ‘known known’ category? This falls squarely in that. Someone, somewhere attacked someone else and someone analyzed that and determined the IoCs. How is it a threat to your organization? Remember, our purpose is ‘Threat’ Hunting, and not IoC hunting? When does an IoC become a threat? These are not easy questions, but it pays if you can answer it before conducting the Threat Hunt. It does make sense when the IoC pertains to exploiting some known vulnerability that exists in your organization and then you can verify if the vulnerability has been exploited in the past by some malicious actor.
This approach also makes sense as a start to Threat Hunting. The hunt can start by looking for relevant IoCs in historical logs and if any is found, look for evidence of compromise. Only then it becomes a threat hunt. If any evidence of compromise is detected, an alert can be generated for further DFIR steps.
It is important to remember though, that this methodology will be least effective in detecting modern targeted attacks. In a recent attack, a large enterprise was subjected to a successful whaling attack. The attacker email was a valid email and actually belonged to CEO of another company. No Threat Intel sources flagged it as malicious. Our investigation revealed that it was part of multiple leaked credential dumps. That’s how the real attackers got hold of a valid email and used that for whaling attack. There was no IoC here. In another instance of a ransomware attack, I saw attackers create brand new domains and used those for Command and Control. Again, no IoC. Hence, the real Threat Hunting has to look beyond IoC search. Searching for IoC do NOT constitute threat hunting.
Any good SOC worth it’s salt should be searching for relevant IoCs all the time. Just do not call it Threat Hunting and assume you are well defended.
Hypothesis Based Hunting: This is the real deal. We know something and use that assumption (or hypothesis) to look for evidence of compromise. This could be based on attacker TTPs (can be based on MITRE ATT&CK framework) for behavior based hunting or on statistical techniques for anomaly based hunting. In this case, we look for Indicators of Attack (IoA) rather than IoC.
Another interesting term gaining prominence is Behavioral Based Indicators of Compromise (BIOCs) that look for attacker behaviors. Does not matter which attacker or what the attack tool, but if they exhibit any behavior akin to malicious act, the BIOC should be able to hunt that down. These BIOCs could be based on process trees, network traffic, file manipulations or modifications etc. If you explore the MITRE ATT&CK framework, they have done a phenomenal service by documenting many known attacker groups and listed out the tools used by them. A closer examination would reveal a small subset of tools and techniques used in many attacks such as Cobalt Strike, Mimikatz, Empire etc. If we hunt for evidence of use of these tools in our environment, we would be able to detect any malicious actor who uses these tools.
Intel based hunting are easy to conduct but are likely to miss the real threat to your organizations. Hypothesis based hunts are much more difficult but are likely to catch even targeted attacks or hitherto unknown threats.
So, how do we conduct Threat Hunting? Let us explore that in the next section.
There are many ways to conduct threat hunt and will probably require a full book. However, it should start with a framework and lead to improvement on SOC maturity. These are the broad steps I would recommend:
- Know your attack surface and vulnerabilities
- Know your real time threat detection capabilities
- Identify gaps in detection vis-à-vis threats. A great way to do it is by using a framework such as MITRE ATT&CK and use tool such as the ATT&CK Navigator to overlay threats with detection as nicely described in the article below.
DeTT&CT: Mapping your Blue Team to MITRE ATT&CK™ — MB Secure
- Conduct hunts for those identified gap areas and leverage these hunts for improved detection capabilities in future.
There are many awesome resources and guides out there as a simple Google search for Threat Hunting would reveal. However, I personally like these:
- Awesome Threat Detection and Hunting collection at Github
- The Threat Hunter Playbook
- I also like the 6 hour free training by Chris Brenton from Active Countermeasures. The course is awesome and conducted free every few months. The next one is in July 2021. There is a more comprehensive 16 hour version also which gets into more details.
Everything in life has a purpose. What is it for Threat Hunting? I guess there are two primary purposes; one, to proactively search for any threats as discussed above and the other, for continuous improvement of real time detection capability.
As depicted above, the outcome of Threat Hunting is to proactively look for hidden threats in your organization as well as contribute to the real time threat detection platform.
It is important to view hunting as a process, with many variables and unknowns; and hence it is primarily analyst driven. Yes, some parts of threat hunt can be automated, especially the data gathering and initial processing/ enrichment, However, the analysis part is typically manual. As a result, picking the right team for hunts is critical. The desired threat-hunter skills should include:
- Systems knowledge (end points, networks)
- Security knowledge (knowledge of threats, vulnerabilities, attacker behaviors, log analysis skills)
- Data analysis (creating charts and dashboards, data analytics, statistical techniques)
- Ability to think out-of-box (curiosity)
A good threat hunt team should include all the above skills — security expert, data analyst and creative thinker. This is difficult to find in abundance. However, these are skills and can be taught to good SOC Analysts and Incident Responders allowing a training pipeline. In larger SOCs, L2/L3 analysts would also make a good candidate to build up threat hunting.
It is important to remember that threat hunting as a practice will mature over time as more and more hunts are added to the hunt library. We should not expect immediate results. It is also important to know that not every hunt will result in detection. When that happens, we cannot assume that our organization is safe. It is just that this current hypothesis or IoC was not validated. Even in this case, we should look to see how this hunt can help improve real time threat detection in future by adding new SIEM rules. A word of caution here, not every hunt is amenable for converting to SIEM rules. Many of them, based on statistics and data analytics cannot be converted to rules. In those cases, explore if dashboards or other visualizations can be created that can help with easier detections in future.
Points to Ponder
- Hunting is people and process driven. We need to identify the right people and develop processes.
- Threat hunting is not tools driven. Too much reliance on tools or a particular data type will result in sub optimal hunt outcomes. Hunts have to be flexible with the ability to pivot around critical data points and pull in additional data as required.
- Outcome of hunt will cause additional load on incident responders in case of positive hunts and for SIEM administrators in case of positive or negative hunt (for creating new rules, if applicable).
- It is not a science but art, and hence the outcomes are difficult to measure. Be prepared for no results or disappointments.
- Threat hunts are good only if the detected alerts are fed into the DFIR pipeline and taken to a reasonable conclusion. Otherwise it is just another alert making machine.
To conclude, threat hunting is an advanced SOC capability that can uncover hitherto unknown threats that are likely missed by the real time detection tools or advanced targeted attacks. However, it should not be taken lightly and SOCs should attempt to reach a minimum maturity level with people, process and tools before attempting to introduce threat hunting. Otherwise, it will only add to the chaos. Threat hunting is also about data and telemetry and hence the capability to pull in right data for hunts is important.
When done correctly, it is awesome though!
