Data Tiering & Replay

Welcome to Data Tiering Visualized

In this demo, we'll explore how to implement an effective data tiering strategy using Cribl. With Cribl's flexible platform, you can optimize data storage, access, and use across a variety of environments, from critical logs to long-term data archival. As a result, you are able to access your data wherever you need it, while also managing costs and improving the performance of your systems.

Tier 1 - Critical Logs

In this demo, tier 1 logs are firewall logs coming from our Palo Alto firewall over syslog. These go to three places: cold storage (uncooked), warm storage (cooked), our SIEM (cooked).

Tier 1 data / logs are crucial for real-time monitoring, alerting, and incident response. These logs are often related to critical system errors, security breaches, or service failures and when immediate access is required.

• Access – Frequent

• Storage – High-availability, low-latency systems for real-time access.

• Retention – Short-term, for immediate detection and resolution; may be archived for critical incidents.

Tier 2 - Operational Data

This demo uses metrics and system state from an agent (in our case, Cribl Edge) for operational data. We route this data to cold storage and a time series database (TSDB).

Tier 2 data / logs provide insights into the daily operations of the system, such as user activities, system events, or API calls. While requiring continuous access, it's not prioritized over critical logs.

• Access – Frequent

• Storage – Medium-performance systems, balancing cost and access times.

• Retention – Medium-term, for troubleshooting, performance analysis, or capacity planning.

Tier 3 - Audit / Compliance / Archival

Our short demo has two main data sources the both send data to long-term (cold) storage. However, cold storage can serve as a repository for all data generated in your environment. Ideally this is a low-cost object store with infrequent access requirements.

Tier 3 data and logs track changes and access patterns, especially important for regulatory compliance, security audits, or forensic analysis. Or Older logs that might not be immediately necessary but are kept for historical analysis, long-term trends, or backup purposes.

Tier 3 Audit / Compliance

• Access – Infrequent

• Storage – Cost-effective solutions; real-time access not needed.

• Retention – Long-term, often due to legal or regulatory requirements.

Tier 3 Compliance / Archival

• Access – Seldom

• Storage – For historical analysis, long-term trends, or backup.

• Retention – Varies; may extend for years.

Datasets Overview

Retention requirements vary based on the type of data. Cribl Lake (below) has four datasets that correspond to our different data tiers.

Tier 1 - Critical Logs

These are the exact same logs that are being sent to our SIEM, however we can store them for 60 days at lower cost. This also clears up our SIEM to run faster, since we don't store as much mid-term data in it.

This dataset also has two accelerated fields: host and sourcetype. This makes Cribl Search more performant when querying this dataset.

Tier 1 - Critical Logs pt.2

Using Cribl Search and Cribl Lake, we can create an incident response dataset to hold logs that have recently been defined as critical. When we're ready, we can send these directly to our SIEM for further investigation.

Tier 2 - Operational Data

While we are sending metrics like system state to our TSDB, those only stay for a few days due to their ephemeral nature. However, keeping tier 2 logs in a medium length data store enables security practitioners to quickly search them in case of incident response. And yes, we can send them directly to the incident_response dataset.

Tier 3 - Compliance / Archive

And for everything else, there's cold storage. Note the 5 year retention period.

Don't forget to (re)hydrate

Rehydration is a large part of data tiering. Being able to access different tiers is half the battle (so is knowledge). Here we can use Cribl Search to grab specific data from the palo_warm dataset and then send it back through Stream to shining SIEM.

Making data easy to move vastly improves your data strategy.

Incident Response pt.1

Looks like we found a host that we want to investigate further. Luckily we kept the metrics in our tier 2 storage and can now quickly send the relevant data to our incident response team.

Incident Response pt.2

Remember that tier 3 archive? Well we can tap that as well for older (60d+) metrics to help search for patterns and better understand the extent of the security incident.

Wrap Up

We've demonstrated how Cribl facilitates efficient data tiering, optimizing data movement, storage, and access. Whether you need real-time access to critical logs, troubleshooting data, or archival data for long-term retention, Cribl gives you the power to optimize your costs, performance, and compliance.

By integrating Cribl into your data workflow, you can ensure your data is always available when needed, while keeping your storage strategy agile and scalable.

Feel free to shedule a demo or try cribl by clicking on either.