Imagine you’re the protector of a castle. Your walls are tall, the gates are strong, and the guards are well-trained. But what if an intruder was still able to slip past your defenses? Even with the best security tools, not every threat will be caught. Threat hunting is the proactive approach to finding attackers that might have bypassed your defenses. Instead of waiting for an alarm to go off, you’re actively searching for signs of trouble, looking for subtle clues that something might be wrong.

At its core, threat hunting is like digital detective work. Your curiosity leads you to sift through mountains of data, looking for anything that seems unusual or out of place. The goal is to uncover threats that might not trigger typical security alerts.

Rather than relying solely on automated tools like intrusion detection/prevention systems or firewalls, threat hunters take a hands-on approach by crafting queries, diving deep into logs, and using their knowledge of potential threats to uncover suspicious activity.

Threat Hunting 101

Let’s break it down. If you’re new to threat hunting, these basics should give you a good foundation.

Getting the Data

Before getting started with threat hunting, the first step is to ensure you have access to the right data. Threat hunting relies on collecting and being able to search logs that cover a wide range of systems: network traffic, authentication logs, cloud infrastructure activity, and more. There is often a challenge of gathering the data efficiently and ensuring you’re not overwhelmed by logs that are just noise, but Cribl Stream can help overcome those challenges.

Cribl Stream can help simplify the process of collecting logs from various sources by enabling you to easily get data across platforms from any source to any destination. The data routing and filtering can also help ensure that only important logs reach your search platform to cut through the noise and improve your threat-hunting process.

Understanding Your Environment

Once you have the logs, the next step is to establish a baseline of “normal” activity in your environment. This involves identifying typical patterns of behavior across your systems, networks, and users. Understanding what’s normal allows you to quickly detect deviations, which could be signs of malicious activity.

Building this mental model of normal behavior requires time and consistent observation. Cribl Search makes this easier by helping you find and visualize regular data patterns. By regularly querying this data, you can gain insights into how your environment behaves under normal conditions, which makes it easier to identify anomalies that could indicate potential threats.

Searching Through Data

Now that you have the data and have a solid understanding of your environment, it’s time to start looking for anomalies. This involves sifting through various logs, such as network traffic, authentication events, application logs, and system activity to identify possible indicators of compromise (IOCs).

With Cribl Search’s query capabilities, you can quickly scan through log data to identify potential security incidents. Running queries regularly can help you build and maintain your mental model of normal behavior so you’re more capable of spotting deviations that might indicate threats.

Analyzing Suspicious Activity

If you find something unusual, don’t panic! It doesn’t always mean your system is compromised. Part of threat hunting is analyzing your data to determine if it’s a legitimate threat or just unusual behavior.

With Cribl Search, you can refine your queries, filter results, and focus on relevant log data to investigate potential incidents more thoroughly. This allows you to validate threats effectively and decide whether action is necessary.

A Threat Hunt with Cribl Search

Recently, our security team was threat hunting and noticed a failure spike in our auth provider logs. We will walk through finding a potential IOC in our environment and tracing it to a source. The screenshots in this section are recreations in a lab environment with redacted information. The intent is to show the process without leaking potentially sensitive information because security and privacy are essential to Cribl.

As we go through our threat hunting process, we use various search commands. If you want more information about the commands in the screenshots, check out the search command docs.

We started our hunt by checking for different types of auth failure events over a 1-hour span of time using the timestats command.

This presented a chart that helped us see a spike in failures and showed us where to focus our efforts. We then checked which exact data type had the most failures by using the count command combined with our time stats chart to start narrowing our search to a specific data type.

Once we knew the data type and timeframe to search, we looked at the actual logs during that time.

In the screenshot below, we’re using the project and render commands to only show the fields we want to see and display it in a way that makes it easier to quickly read. If you know the exact fields you want, this can be helpful to get rid of the data that might just be noise for hunting purposes. As you can see, the log data revealed possible fuzzing activity, which required further analysis to determine whether any of the payloads were successful.

Our search also revealed a source IP, and we were able to use the ip-lookup command to get location information, which can help determine where the requests originate.

We then took that IP address and did a wider search to see what other activity we could find. The goal now was to see if we could find a username associated with the IP, and fortunately, we were able to find one.

After a quick search on the username, we determined that the user is a security researcher through Bug Crowd. The researcher was researching ethical vulnerability and hoping to find something to report through our public vulnerability disclosure program. Unfortunately for our researcher, none of the testing produced anything to report. Still, we appreciate their efforts and hope they keep trying and promptly and responsibly report any vulnerability findings to us.

Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.

We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.