We re-architected our federated search engine from the ground up - and out of the gate, it's already 3–5× faster, far more efficient, and we're just getting started.
Coming off the heels of our very exciting announcement of a new lakehouse engine in Cribl Search, I’m also excited to share that we’ve amped up the performance of the original federated engine to run significantly faster searches. Now supported on our most popular data sources and formats, including Amazon S3, Azure Blob, and Cribl Lake datasets, and Parquet and NDJSON file formats.
No more tradeoffs
Imagine it's 2 am and a critical system has failed. Your monitoring system has rolled the logs, so the data you need is sitting in S3. You initiate a query and wait. Before the results arrive, you've already mobilized a response team, summoned additional experts, and begun the arduous process of migrating data just to facilitate a search.
That's the "tax" of searching object storage — and it forces today's compromise:
Fast but expensive systems for real-time investigations
Slow but cheap storage for everything else
As a result, Investigations stall when data isn't in the "right" place, forcing teams to duplicate data just to make it searchable. Costs climb as volumes grow, and analysts end up waiting for results instead of answering questions.
In the moments that matter most, that's not acceptable. We rebuilt our engine to eliminate that compromise. Cribl Search v1 was our answer to that compromise: query where the data lives instead of copying it everywhere. It proved that search-in-place was a real path forward — that you didn't have to move your data into a separate system to ask meaningful questions about it.
A better approach: Faster federated search
While v1 was good enough for secondary use cases — compliance audits, historical forensics, the occasional needle-in-a-haystack query where latency didn't really matter — it wasn't something you'd reach for at 2 am with the clock running.
We wanted Cribl Search to be fast enough that you could reasonably use it for live investigations — without abandoning the economic benefits of keeping data in object stores. That meant a different design, not a faster version of the old one. So we made the call to rebuild rather than tune.
Introducing Cribl Search v2
Search v2 is a complete rewrite of our federated engine. We took the lessons we learned in v1 and rebuilt the engine on a more efficient, more scalable design — modern, multi-threaded execution that does more work in parallel and moves far less data around to get there.
You don’t have to learn anything new or change how you work. Search v2 is already 3–5× faster than the best v1 could deliver and we are just getting started.
With Search v2 now supporting Parquet, the two finally come together: a columnar format built to skip what your query doesn't need, on an engine fast enough to make that skip pay off in your search wall-clock time.
If you're evaluating Cribl Search, think of v2 as the same experience you already know — just materially faster and more economical on object storage. If you're already running Search, v2 is your on-ramp to the same answers in less time.
Either way, the core idea is simple: search-in-place shouldn't feel like a last resort. With Search v2, your object storage can finally pull its weight in the moments that matter most. To get started with v2, see Cribl Docs.
