The case for running detections in the data pipeline has become much louder. The promise is appealing: inspect events while they are moving, alert within seconds, and respond before an attacker has time to spread. CrowdStrike has been making that case since acquiring Onum, and I hear versions of it in customer conversations all the time.
I understand the appeal. I co-founded a pipeline company, and for eight years we have argued that pipelines should do much more than move bytes from one place to another. There is real value in processing security data before it lands. But I think the broader claim collapses several different jobs into one. A rule can detect something in a single event. Turning that event into an alert worth acting on usually requires correlation, and that distinction will shape how security teams build over the next decade.
My view comes from operating these systems as well as building them. Early in my career, I ran systems that billed and processed a billion dollars a year and carried the pager when they broke. I later built products at a SIEM company. I have been woken up at 3am by alerts that did not matter, and I know what repeated false positives do to the people expected to act on them.
Where the in-stream argument is right
Speed matters more than it used to. CrowdStrike's 2026 Global Threat Report puts average eCrime breakout time at 29 minutes and the fastest observed breakout at 27 seconds. A detection loop that batches data, indexes it, waits for a scheduled search, and then pages a human can waste a meaningful part of that window.
The economic criticism is also fair. Paying to ingest everything before deciding what matters creates a tax on curiosity. That pricing model is one reason this category exists and one reason we built this company. A data layer should be designed around customer needs rather than the analytics vendor's pricing model.
Security data should be parsed, enriched, shaped, scored, and routed while it is moving. The harder question is how event-level detections become alerts a SOC can trust.
What reliable detection requires
Many detections do run against single events, and that is often the right way to produce an early signal. But the output a SOC acts on is an alert. Most alerts, and certainly most alerts worth automating, need correlation of some kind. Sometimes that means joining events across sources or a time window. Sometimes it means combining one event with identity, asset, or historical context. In either case, the correlation is what establishes that the event matters.
I evaluate an alert by looking at who acts on it and what happens when it is wrong. An analyst may spend twenty minutes investigating it. An automated system may isolate a host or disable an account. If the queue is noisy, analysts learn to distrust it. If a noisy rule is connected to response, the cost can be a production outage.
Correlation is what turns those individual detections into useful alerts. A failed login by itself tells you very little. Fifty failed logins in two minutes followed by a success is different, and recognizing the difference requires state. PowerShell launched from a Word document by a user who has never used it, on a crown-jewel system, requires enrichment, a baseline, and asset context. A command such as net group "Domain Admins" may run legitimately thousands of times a day and become interesting only as part of a sequence.
CrowdStrike's own report says that 82% of detections last year were malware-free. Attackers increasingly use legitimate tools and valid credentials, so the individual events often look ordinary. The sequence, the actor, and the surrounding context are what make the activity malicious. An engine that evaluates each event independently can produce detections, but it cannot turn many of them into high-confidence alerts without correlating the events with each other and with their context.
The Onum acquisition announcement describes a stateless, in-memory architecture. Stateless processing is excellent for throughput, but most high-fidelity detection logic is stateful. It relies on windows, thresholds, joins, baselines, and risk accumulated across events and sources.
Detection engineers have already worked through this problem. Community Sigma rules are often marked experimental and include known false-positive cases because raw single-event matching does not hold up well at scale. Sigma later added a Correlations specification so teams could express the stateful detections they actually needed. Moving the same match earlier in the data path does not solve that limitation.
What IDS and EDR taught us
In-stream detection has a long history. Snort shipped in 1998, and mature security teams learned to treat IDS output as one signal feeding a correlation layer. Raw signature matches can be useful telemetry, but paging people on them directly creates more noise than confidence.
EDR is another useful precedent. EDR products earned the ability to quarantine automatically because an endpoint sensor has deep local context, including process trees, memory, and file lineage. Those products also use cloud analysis and logic tuned across millions of endpoints. That depth of context supports automated response. A match over a syslog event does not provide the same assurance simply because it runs quickly.
This matters even more as teams add AI-driven response. An AI SOC analyst starts with an alert and then gathers history, identity data, asset criticality, related events, and operational context such as a recent change ticket. Much of that information comes from stored data. The correlation still has to happen, and an agent may explore far more hypotheses than a human analyst has time to check. That increases the value of accessible history.
The work before and after deployment
Before a serious detection team deploys a rule, it backtests the rule against historical data, measures what would have fired, tunes it, and repeats the process. When new intelligence arrives, such as a C2 domain or an IOC from an ISAC bulletin, the team also needs to search backward because the attacker may have been present for weeks.
A stream only sees an event as it passes. Backtesting, retroactive hunting, and incident investigation all require full-fidelity data at rest. Vendors offering in-pipeline detection generally still send data downstream to a SIEM or lake for that reason. The pipeline can identify some signals sooner, but the rest of the detection and investigation workflow still depends on stored history.
I have said for years that you cannot ask questions of data you do not have. High-fidelity telemetry may sit unused for long periods and become essential during an investigation. Keeping it is now much more practical. Object storage offers retention at a fraction of indexed-storage cost. Lakehouse architectures make that data fast enough for interactive queries, and federated search lets teams examine data where it already lives without moving and re-indexing every copy. What was once a luxury is becoming standard infrastructure for detection engineering, incident response, and agentic investigation.
How I would divide the architecture
The data layer has substantial work to do on both sides of this boundary.
While data is moving, the pipeline should add threat intelligence, identity, asset, and geographic context. It should run the small set of atomic tripwires that are reliable from a single event, including known-bad IOC matches, canary tokens, and events that are impossible by policy. For everything else, it should produce enriched signals with enough context and provenance for downstream correlation. It should score and route important data into the hot tier of a detection engine. It can also reduce volume when the architecture includes replay, so data filtered as noise today can be recovered if it becomes evidence later.
At rest, teams should retain full-fidelity history in storage they control. That history lets the alerting layer correlate signals across sources and time, and it supports backtesting, retroactive hunting, baselines, and investigations that turn one alert into many questions. Federated search should let analysts and agents follow those questions across lakes, SIEMs, and other stores without forcing every system into one vendor's platform.
The pipeline can run single-event detections and produce fast, enriched signals. The alerting layer then correlates those signals with stored history and other sources before asking people or machines to act. That division gives teams both speed and confidence.
The platform question
There is also a strategic issue worth considering. When a single-platform vendor offers detections in the pipeline, ask whose detection content runs there and where the pipeline sends the data by default. If both answers point to the same vendor's SIEM, the architecture creates a strong pull toward that platform. That may be an acceptable choice, but customers should make it deliberately.
The data layer is a long-lived architecture decision because every analytics engine, AI agent, and future tool depends on it. I want that layer to expand a customer's options. A team should be able to change detection engines without losing its history, give several tools access to one controlled copy of the data, and search across systems without becoming trapped inside one vendor's store. A customer can value a vendor's detections and still retain control of the underlying data and the freedom to change what gets built on top of it.
Questions to ask any vendor selling in-stream detections
Can I backtest a rule against historical data before deploying it, and how much history is available?
When new threat intelligence arrives, how do I search data the stream has already processed?
Can you show me a cross-source, time-windowed correlation running in the pipeline rather than a single-event match?
When a rule becomes noisy, what is the tuning workflow and who carries the operational burden?
Whose detection content runs here, where does the data route by default, and what happens if I change SIEMs?
Conclusion
A large share of detections can and should run against individual events. Most alerts still need correlation before a person or machine can trust them. The architecture has to support both: fast signal generation while data is moving, and the context, testing, and investigative depth that come from stored history. Security teams need both, built on a data layer they control.
