Frozen Is Only Good for Margaritas and Elsa, Not Your Critical Data!
It’s not new news that organizations are producing more data than ever. But, in order to take advantage of this data, it needs to be collected, stored, retained, and then, at some point, analyzed. Most analysis tools also act as the retention point for this data. While this may (at first) appear to be the best option for performance, it quickly creates significant problems. First, those systems were never designed for the scale of today’s growing volume of data, currently at a 28% CAGR. Second, analysis systems pricing is based on the volume of ingested data – just what is sitting there, even if it is not providing value. Finally, those license costs aren’t cheap (and can be prohibitive) and will just continue to climb, exceeding any budget.
Organizations started using the “frozen bucket” method as the data volumes grew and aged out. This allowed them to archive data from their analysis environment, helping to improve performance on live data, but only solved half the retention problem. It saved some space and money. But, pulling that ‘frozen data’ from the archives required a “thaw” process of entire files to make them available to the analysis system. While this wasn’t technically difficult, it was time-consuming, often involving long SLAs and additional ingestion costs. Additionally, many systems didn’t provide the capability to limit the data you were extracting selectively – it was everything in the “frozen” file or nothing.
Let me give you a real-life example we experienced. As Joe Friday said, “The names were changed to protect the innocent.” Actually, it’s to protect the guilty and keep our legal team off my back.
A large customer of ours had an IOC (an indicator of compromise) and needed to retrieve archived data to backtrace the event as part of their investigation. With their existing system, they would need to retrieve two weeks of data, approx. 26 TB. Their vendor told them it would take 24 hours (note: that was a part of the original SLA agreed to). The customer didn’t want to wait that long to address the issue, so they contacted our Product Team. It went something like this: “… you guys keep making all these promises about the capabilities of Search, now show me what you can do for real. Kind of a put up or shut up moment.” Oh, did I mention this was on a Friday night? No, really, it was.
So, the team started digging through the data. First, they realized the data they needed to examine was distributed across multiple regions – no problem! Cribl Search supports Federated search capabilities and can simultaneously query multiple data stores. Then, through some trial and error, they were able to locate the specific datasets in question. It was only a subset of the original dates, three days vs two weeks.
All of the above took about 1 hour between Cribl and the user. At this point, it was a simple matter of targeting the specific data, searching the data where it was, and then only retrieving the data of interest. It was only about 1.2 million events, not the original 26 TB. This data was then retrieved, had a little shaping, and was sent into the existing system of analysis for additional analysis – a total time of 1.5 hours. As you can expect, the customer was very happy with Cribl.
The bottom line is that with Amazon S3 and other similar cloud object storage offering pricing of pennies/GB per month, storing data in these systems and with restrictions on how it is archived and retrieved no longer makes sense. The best practice is to have the ability to separate your system of retention from your system of analysis. Basically, put your data wherever you want in a separate, cost-effective repository (like Amazon S3). Then, optimize the transfer of only specific datasets from storage into your analysis system instead of retrieving (umping) everything back into your analysis system. THAT is where Cribl Search shines.
So What is Cribl Search?
Cribl Search is an innovative new approach to finding and accessing data regardless of where it is landed and in any format. As users embrace tiered data strategies and the reality of multiple analytics and security tools, Search provides a federated solution built to separate the query engine from a storage medium. This delivers a unified query interface in a familiar and ergonomic pipe-delimited language that reaches into existing object stores filled with messy, unstructured, or structured datasets. It retrieves data without moving it or having to index it first. In addition, the same interface can also connect to APIs, databases, or existing tooling and join together results from all these disparate datasets in comprehensive dashboards, scheduled searches, and alerts.
The power of Cribl Search lies not only in what datasets it can reach but also in its ability to discover and forward critical data to your systems of analysis with surgical precision. Targeting specific datasets helps avoid the cost of expensive storage inside a system of analysis. Thus increasing users’ scope of analysis without needing to ship, ingest, and store the data first. Plus, providing relevant, valuable data that are only routed for further analysis if necessary.
Suppose you always had a full-fidelity copy of your logs, metrics, and traces in Amazon S3. Ask yourself this: Would you still bring every event into your analytics systems? Would you truly need to keep terabytes of noisy, verbose, hard-to-search logs in your expensive analysis tools daily?
Wrap Up
Data volumes are enormous and growing, but budgets are not. The percentage of data being analyzed will continue to drop due to licensing costs, giving organizations only two options to address this: get a bigger budget or be more intentional about how data is processed before ingesting into the analysis system. Cribl Search is a true game changer. You can now effortlessly identify and then collect specific datasets and forward them to different systems for advanced analysis, audit, and compliance. It is a tremendous value for anyone managing digital exhaust data at scale. By separating your system of retention from systems of analysis, you can optimize your budget in ways not previously possible. And, since the data is archived in the format of your choice, you’re free to use it however you’d like.
Key Cribl Benefits of Cribl’s Solutions:
- Enables separation of the system of analysis from the system of retention
- Store raw data in low-cost data stores, not in expensive analysis systems (<$$)
- Query data in-place (data stores), then route only relevant data (wheat from chaff)
- Forward just the relevant data to the system of analysis (lower the ingest license)
- No waiting for vendors to have to ‘thaw’ your data
- Improve the quality and speed of your analytics environment by saving older data somewhere else
- Use Cribl Search to front-end and complement your existing analysis tooling
- Keep more data for longer retention periods and pay a lot less
- Replay data to any analytics tools for unexpected investigations
Ready to learn more about Cribl Search?
Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.
We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.
