IT and security teams are expected to move faster than ever. They need to investigate more telemetry, more often, without adding more infrastructure or operational complexity.
When we first launched Cribl Search, we introduced the first ever federated search engine that let customers search across data wherever it lived—without moving or rehydrating it. It was a fundamentally different approach to search, and our customers quickly saw the power of being able to reach across their entire telemetry environment from a single experience.
But as that reach expanded, so did their expectations. IT and security teams wanted to search more data, more often, and get answers faster…without relying on their costly SIEMs.
So we built a second engine that works differently.
The Lakehouse Engine brings compute closer to the data, combining ingest, storage, and query processing in a Search-native architecture. Instead of moving data into a separate analytics platform before it can be useful, teams can send data directly into Cribl Search, query it immediately, and use the same unified experience for dashboards, notebooks, alerts, and investigations.
The result is two engines, purpose-built for different needs: Federated Engine for reaching data wherever it lives, and the Lakehouse Engine for high-performance search over data managed directly in Cribl.
Lakehouse Engine in a nutshell
A Lakehouse Engine is a storage-plus-compute unit that ingests, stores, and accelerates data inside Cribl Search. It backs Search’s direct-ingest workflow and Search Datasets, keeping ingested telemetry in search-optimized storage for fast, schema-aware queries and AI-assisted analysis.
That architecture is different from Cribl Search’s Federated Engine, which provides compute for querying data stored in external systems without ingesting it into a Lakehouse Engine. The two modes share the Search interface and query experience, but they solve different data-access problems.
The technical architecture: ingest, structure, store, search
The Lakehouse Engine simplifies the path from incoming telemetry to an investigation-ready dataset:
A supported Search Source receives streaming data directly through endpoints such as Syslog, Splunk HEC, OpenTelemetry, or HTTP. (This data can be received from Cribl Stream or sent directly to Cribl Search without going through Stream.)
Datatypes interpret the incoming data, and extract fields that can be queried. Search supports Auto-Datatyping with AI as well as custom datatype rules.
Dataset Rules route parsed events into one or more Search Datasets, which specifies a particular Lakehouse Engine and a retention period.
The Lakehouse Engine stores the resulting data and provides the compute that executes queries over those Search Datasets.
Search, Notebooks, Dashboards, alerts, and AI-assisted workflows use the same underlying datasets through a unified interface.
This creates an ingest-to-investigate path without requiring a separate Stream-to-Lake-to-Lakehouse chain for customers whose data can be accepted as-is. Cribl Stream and Cribl Edge remain available when a workflow needs collection, transformation, enrichment, redaction, filtering, or routing before storage. Learn more about getting data into Cribl Search.
What happens when data enters the engine?
A Lakehouse Engine is not just a object-store bucket attached to a query interface. It is the capacity boundary for a Search workload: it receives data, stores it, and supplies the compute used to search that data.
When data arrives, Search automatically parses each event into structured fields using Datatypes. That structure is important because it gives queries known field names to work with rather than forcing every analyst or downstream workflow to rediscover event structure at query time. Datatypes can recognize common event formats, break raw input into events, and turn event content into fields. Search’s v2 Datatypes also support Auto-Datatyping and formats such as newline-delimited JSON, delimited text, JSON arrays, key-value pairs, Parquet, raw text, and XML, depending on the source and provider. Learn more about datatypes in Cribl Search and Federated Search v2.
After parsing, Dataset Rules determine the Dataset that stores each event. This separates data organization from the source itself: teams can route different event types, environments, or retention classes into Datasets that match their operational and governance needs.
Search Datasets provide the logical data layer
Search Datasets are the logical collections analysts query. Each dataset is backed by a Lakehouse Engine and has its own retention setting, allowing teams to manage data lifecycle at a more granular level than setting one retention period for an entire engine.
Retention can range from one day to ten years. When a dataset reaches the end of its retention period, Cribl Search deletes the data. This lets teams align storage duration with access patterns and business requirements—for example, keeping high-value application or security telemetry available for active investigations while retaining other data for longer forensic or compliance windows. Learn more about setting up a Lakehouse Engine.
The engine provides the physical capacity; Search Datasets provide the logical organization and lifecycle controls. That separation helps teams allocate resources by workload while keeping search and retention decisions understandable to the people who manage the data.
Sizing: capacity is based on ingest and workload
Lakehouse Engine sizing begins with the amount of raw, uncompressed data expected per day, with compute capacity reserved for Searching into that data. Available engine tier sizes range from Nano through 6X-Large. If ingest patterns change or a team sees ingest or search latency, an administrator can resize an engine or add another engine to distribute workloads. Learn more about sizing a Lakehouse Engine.
An important point is that while you can size an engine larger than necessary, you shouldn’t make it smaller than the minimum specified by the ingest performance. Search will generally benefit from having a larger Lakehouse Engine specified with faster-performing searches; however, if too small a Lakehouse Engine tier is chosen, it may not be able to keep up with ingest.
The engine’s ingest rate is measured at the Search interface, before any transformations are applied to the data. In practice, this means the engine counts the bytes it receives—not the original volume before Stream or Edge changes the event. Parsing, enrichment, added fields, or retaining a copy of _raw can make the payload received by the engine larger than the original source volume.
Cribl Search compresses ingested data at rest. The documentation describes a typical compression ratio of approximately 10:1 to 12:1, although the actual ratio depends on event shape and content. Storage scales with the compressed data retained and the retention periods configured on the datasets.
This model separates two operational decisions:
Engine size determines the daily ingest capacity and the compute envelope for the workload.
Dataset retention determines how much compressed data remains available over time.
Together, these controls let teams size for throughput and query demand while tuning retention by dataset rather than applying one broad policy to everything.
Why ingest data into a Lakehouse Engine?
The Lakehouse Engine is designed for telemetry that benefits from repeatable, low-latency access. Typical examples include application logs, firewall events, endpoint telemetry, infrastructure logs, and other data used regularly for incident response, root-cause analysis, threat hunting, dashboards, and operational reporting.
Because the data is already stored inside Search and organized into structured Search Datasets, analysts do not need to rehydrate or re-index it before beginning an investigation. The same datasets can support repeated queries, visualizations, scheduled searches, alerts, Notebooks, and AI-guided workflows.
This is particularly valuable for iterative analysis. An investigator can start with a broad question, narrow the time range, pivot on fields, compare related signals, and save the investigation without turning each step into a separate data-movement project.
Direct, simplified ingest with AI-powered parsing
Direct ingest is a lightweight option for customers that want data searchable quickly and largely as-is. Search can receive supported sources directly, automatically parse incoming events, and route them to Search Datasets.
That does not make Stream or Edge obsolete. The distinction is architectural:
Search direct ingest is optimized for speed and simplicity when data does not need extensive preprocessing.
Stream provides advanced processing and routing when teams need to transform, enrich, reduce, redact, normalize, or send data to multiple destinations.
Edge remains useful for collection close to endpoints and infrastructure.
Customers can use these products together. They can also send data directly to Search when the added pipeline control is unnecessary. This gives teams a choice between a short path to search and a more programmable data-processing path.
Lakehouse Engine versus Federated Engine
Cribl Search is powered by two engines:
The choice does not need to be all or nothing. A practical tiering strategy is to place high-value, frequently accessed telemetry in a Lakehouse Engine for fast analysis, while leaving high-volume, lower-touch data in its existing location and accessing it through the Federated Engine. Search presents both modes through one interface and supports common investigation workflows across them.
A useful mental model for architects
For architects, the Lakehouse Engine can be viewed as three layers working together:
The ingestion layer accepts supported telemetry and applies datatype interpretation.
The data layer stores structured events in Search Datasets and applies dataset-specific retention.
The execution layer runs schema-aware queries and powers the Search analysis experience.
The important design decision is not simply where data is stored. It is which data deserves a Search-optimized, ingest-based path because the organization expects frequent access, interactive exploration, or high concurrency. Data that does not justify that path can remain in external systems and be queried through federated search.
This dual-engine approach lets teams design a workload-aware architecture instead of forcing every byte into the same platform or storage tier.
Customer value: faster answers with less operational overhead
The Lakehouse Engine helps organizations:
Accelerate investigations by keeping important telemetry ready for fast, repeatable search.
Reduce architecture complexity by combining ingest, storage, and compute in one Search experience.
Shorten time to value through direct ingest, automatic data interpretation, and Search Dataset creation.
Improve query quality with structured fields and schema-aware search rather than repeated manual parsing.
Match retention to business value with dataset-level lifecycle controls.
Support collaborative workflows through Dashboards, Notebooks, alerts, and AI-assisted investigations.
Preserve architectural choice by combining ingest-based search with federated search-in-place.
Relieve SIEM performance pressure by moving general-purpose analytics to a purpose-built search and analytics engine.
Scale shared access across teams or agencies with RBAC and dataset-level segmentation.
Reduce schema-management effort as new teams, sources, and data structures come online through auto-data typing and normalization.
The value is not simply another place to store logs. It is a more direct way to turn telemetry into context and action—without requiring every investigation to begin with a data movement, rehydration, or indexing project.
The foundation for an investigation-first Search experience
Cribl Search’s Lakehouse Engine brings storage and compute closer to the work analysts need to do. Instead of separating collection, storage, indexing, and investigation across multiple systems, it provides a unified foundation for getting data in, making it searchable, and using it to answer operational and security questions.
When teams need fast access to frequently queried telemetry, the Lakehouse Engine provides a structured, Search-native path from ingest to investigation. When data should stay where it is, the Federated Engine extends the same experience to external systems. Together, they give organizations a flexible way to search more data, control operational complexity, and get to answers faster.
