When we created Cribl Search, we wanted to give systems administrators the ability to query data without having to spend resources on collection and processing first — but we didn’t stop there. With Search, we’re also making it possible to query all the data you’ve already collected, processed, and kept in places like object stores, file systems, analytics tools, S3 buckets, or other data stores.
Administrators have to work with tens, hundreds, or even thousands of terabytes of data being generated daily. Some of it gets forwarded to Splunk for processing, some might go to a homegrown system based on Elastic, and some might end up in a file system or other location. But most of this stuff will end up in S3 buckets, because paying to ingest it all into a central system of analysis or store it locally could bankrupt your organization or at least your department.
Storing it in S3 buckets, or similar stores isn’t necessarily a bad thing, but whether or not you’ll actually get around to querying this information is kind of a crapshoot. There’s always a certain amount of data that’s queued up into your system of analysis to be processed. Still, there might be 10x that amount in storage that you need to keep for whatever reason — maybe because of some legal requirements or because it’s your “just in case” data. Either way, when it’s stored in these buckets, the chances of you going back to look at it are slim, so there’s no way to know if there’s anything of value there.
With Search, we’re giving users the ability to query all of the data in those stores without having to move, reformat, or do any processing first. Not only that, you can use it to query the data in multiple data stores simultaneously. We’ve created a federated search tool that can look at each of your storage locations by executing a single query, whether the data is raw or formatted.
Now you won’t have to worry about sucking all your data into a single system of record to be processed. Just leave everything where it is and use Search to interrogate all that data without the cost and complexity of moving it. Keep your data where it is, where it belongs and just move the interesting data you may want to analyze further.
Be able to query your data, wherever it lives, without having to replace any of your existing observability solutions. We’re not looking to become your data store or another data lake. We know you’ve spent a lot of money on these solutions because they address a specific need, so we designed Search to complement and work with these solutions to optimize your environment and improve insights into the data you already have.