But, isn't data, just data? Maybe not!
With Cribl’s introduction of Lakehouse, a powerful new feature of Cribl Lake that allows for fast searches on recent data, we’ve changed how you should approach data storage and analysis. Cribl Lake initially launched as a cost-effective, long-term retention storage solution, but now with Lakehouse, it’s a comprehensive, full-service shop for all your storage needs: archival storage for compliance, fast queries for needle in the haystack searching, and everything in between. Cribl Lake is purpose-built for the dynamic, unpredictable nature of telemetry data, unlike traditional solutions typically focused on cheap storage that won’t fit in your SIEM. Instead, Cribl Lake offers an automated tiered storage solution that optimizes both performance and cost for all data types without compromise.
But What Is Tiered Storage and Why Do I Need It?
Let’s talk about the 3 V’s of telemetry data: Volume, Variety, and Value. The bottom line is data varies- it varies in variety (data types, data formats, etc) and needs to be treated as such, not just for security reasons, not just for governance reasons, but as a general best practice for your data strategy for 2025 and beyond. Beyond variety, it also varies in value. For real-time alerting, its value may diminish in minutes, but for compliance and incident investigation, it may only become valuable months or even years from now. The key is to identify the different data types and potential value, then provide appropriate storage classes to support your IT and Security teams' requirements.
Understanding Data Challenges
If I am on the SecOps team, I need real-time data to alert me of issues; days or even hours old data is of no value in responding to an event. On the other hand, if I am in ITOps, hours, days, months, and years might be exactly what I need for trends or compliance reasons. The worth of data is dependent on its age, accessibility, volume, etc, not to mention who is interacting with it. Consider that some real-time, low volume data might just become the most critical data in a month from now.
And Now a Message from Your Sponsor
Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Powered by a data processing engine purpose-built for IT and Security, Cribl’s product suite is a vendor-agnostic data management solution capable of collecting data from any source, processing billions of events per second, automatically routing data for optimized storage, and analyzing any data, at any time, in any location. With Cribl, IT and Security teams have the choice, control, and flexibility required to adapt to their ever-changing data needs. Cribl’s offerings beyond Lake –Stream, Edge, and Search–are available either as discrete products or as a holistic solution.
Tiered Data - The Basics
Tiered data management is about balancing cost and complexity. As organizations grapple with the challenges of managing IT and security data, many have turned to tiered data management strategies to address these concerns. They are embracing a data management strategy that efficiently delivers data into tiered storage based on timeliness requirements for operational versus exploratory use cases. They’re also accounting for retention requirements for regulatory and compliance needs.This tiered approach involves using a combination of legacy tools, cloud data warehouses, lakehouses, and data lakes to store and analyze data based on its value, usage patterns, and retention requirements. There is no one-size-fits-all approach.
The data engine provides automated and flexible data tiering, allowing users to rationalize their data storage philosophy, aligning where data is stored with its value and usage through data tiering. Performance-optimized data can be delivered to the range of analytics, monitoring, and cybersecurity tools in use, providing the most advantageous data product for those platforms.
Maintaining full-fidelity datasets is key, as is cost-optimized storage. This full-fidelity storage is often required for exploratory or compliance use cases. This tiering strategy allows users to put data in the right location for its desired outcomes and use, aligned to the value of the data. Coupled with the distributed access and governance inherent in the data engine, each tier is accessible regardless of which tier data is stored within.
Why Consider a Tiered Data Structure
If you're still sitting there, dumping everything into your SIEM, your budget is going to bust if it hasn’t already. If you are filtering or sampling data to fit a license limit, you probably think you're missing something important, and if you are dumping SIEM overflow and aged-out data to cold storage to save money, you know it's a real pain to rehydrate that data if needed.
If you have any of the above issues, it’s time to consider a new data lake strategy — a tiered data strategy. It starts with– where does your full fidelity data go? You probably (or should) process it through a pipeline, providing greater data management and cost controls, because now only critical/actionable data goes to your SIEM or other system of analysis. But where does all the rest of that data go? You need to start thinking about a data tiering strategy, where data is structured based on its usage and/or value. If you have infrequently accessed data, you need an archive, but one with real-time visibility and access, without the delay associated with rehydration if you need to investigate it. I need some form of high-speed searching, for when I have those ‘needles in the haystack’ challenges. This means I need some form of columnar storage in order to give me the analytics, performance, and real-time query access that I need to alert on this data dashboard, this data. And I need storage for infrequently accessed data for audits and reports. This is ultimately what Lakehouse is all about. This is the type of storage architecture that you're going to need to be successful with telemetry data management over the next 10 years, whether you use our products or somebody else's.
Cribl’s Lakehouse feature redefines how organizations collect, store, manage, and analyze telemetry data at scale, ensuring a future-proofed, cost-efficient, and flexible approach to data management. By combining the performance of a high-speed query engine with the cost savings of object storage, our new Lakehouse empowers teams to gain instant insights without vendor lock-in or unnecessary storage costs.
The 5 Whys for Implementing a Tiered Data Strategy
1. Store Data Based on Value, Usage, and Access Needs
Reduce costs and have data readily available by routing high-value, frequently accessed data in performance-optimized tiers for real-time analytics. Keep less critical or exploratory data in cost-effective storage.
2. Use Open Formats to Unify Data
Have a single source of truth across all tiers by consolidating fragmented data across various systems. Run analysis without unnecessary duplication or movement between systems.
3. Prioritize Flexible and Scalable Data Analysis
Tiered data management combines on-demand compute with cost-effective storage to analyze large data volumes without continuous infrastructure expansion.
4. Utilize Cost-effective Storage to Meet Compliance and Audit Requirements
Store full-fidelity data in low-cost object storage for long-term retention requirements. Readily retrieve and replay data as audits and investigation needs arise.
5. Reduce Tool Proliferation and Complexity
Consolidate security and observability platforms to reduce reliance on specialized tools, minimize skill set fragmentation, and simplify the data lifecycle.
Summary
Implementing a tiered storage strategy requires understanding the operational, security, and business requirements of the organization and the data it collects. Not all data is of equal value; using data tiers to classify data makes it easier to locate and retrieve specific, relevant, and/or timely information. By separating into data tiers, you can prioritize information for quicker access and processing. Less important or infrequently accessed data can be ‘frozen’ at reduced costs but takes longer to retrieve. User needs and data requirements vary greatly, so structuring data in tiers optimizes access and costs. Proper tools and solutions, like log management systems or SIEMs, will aid in executing this strategy efficiently.
How to Get Started with Lakehouse
Existing Cribl.Cloud User: Getting Started with Lakehouse
New to Cribl.Cloud: Get a FREE cloud account
Play Around: Cribl Lake Sandbox or Cribl Search Sandbox
Learn More about Cribl Lake
Get FREE Training at: Cribl University
Ready to dive deeper into Cribl Lake, Lakehouse, Tiered Storage, or Cribl Search?
Learn more about Cribl Lake, Lakehouse, and Search Here