Master your data, not the platform - og image

Master your data, not the platform

Last edited: November 12, 2026

This is the third post in a series on SIEM migrations. The first post covered why they fail: vendor owned data pipelines, teams without the skills to run something new while keeping the lights on, and contracts written to make leaving expensive. The second post covered the data dependency and how to break it. This post is about the second problem, the skills, and why the answer is not another vendor certification.

Think about the best analyst on your team. The one who finds things nobody else finds. They know which log source never parsed correctly, they know the three fields that actually matter for lateral movement in your environment, and they can go from a vague question to a real answer in about ten minutes.

Now change the platform underneath them.

On day one of the new SIEM, that person is a beginner. Not entirely, but enough that it hurts. The queries do not work. The field names are different. The data model is different. The tuning knobs are in different places and some of them do not exist at all. Four years of hard won speed just got reset, and your best person now has to relearn the basics while still carrying a full caseload.

That is the part nobody budgets for. Leadership sees a platform swap. What is actually happening is that a large chunk of your team's expertise just got written off, and there is no line item for that.

Your analyst's skill was never one thing. It was two things tangled together. One part was knowing how to think about data: what a good question looks like, how to narrow a large set of events down to a small one, when a spike is meaningful and when it is noise. The other part was knowing where the buttons are in one product.

The first part moves with them anywhere. The second part does not. Most security programs never separate the two, so when the platform changes, it all feels like a total loss.

I ended the last post promising a piece on platform fluency. The fluency worth building is not fluency in the platform, it is fluency in the data underneath it, because the platform is only the interface you happen to be looking through this year.

The dependency that lives in people

Post two was about data lock-in, and that one is at least visible. You can draw it. Agents, protocols, storage formats, query languages, all of it shows up on an architecture diagram if you are honest about what you are drawing.

Skills lock-in does not show up anywhere. It lives in people's heads, it never gets documented, and nobody puts it on a risk register. But it is the same trap with the same economics behind it. Vendors run free training, free certifications, and a whole community program because a certified team is a team that argues internally to stay. That is not a conspiracy, it is just good business for them. It only becomes your problem when the certifications are the only thing your team has.

I worked with a hunt team that had exactly this problem. Their hunting practice was genuinely good, and it ran almost entirely through one senior engineer who was excellent in the vendor's query language. Everyone else could read his queries. Nobody else could write them well enough to hunt independently. When he left for a better offer, the hunt program did not degrade, it just stopped. For two quarters, the team ran the saved searches he had written and did not create a single new one, because the barrier to writing a good query in that language was high enough that nobody wanted to be the person who got it wrong.

They eventually rebuilt the practice on top of their data lake in SQL. Within a quarter, four people were contributing hunts instead of one. The people had not changed. The barrier had.

SQL is the example here because it is the most widely supported option, but the point is not SQL specifically. It is choosing a query language that works in more than one place. KQL (Kusto Query Language) is vendor neutral as well. The language is openly specified, the parser is published under an open source license, and anyone can implement against it, which is why it now runs well outside where it started, including in tools like Cribl. Compare either one to something like SPL, where every query you write and every hour your team spends getting good at it stops being worth anything the moment you leave the product that owns it. The language itself matters less than whether it travels.

That is the honest version of the skills problem. It is not that your team is not smart enough to learn a new platform. It is that you concentrated the expertise into a narrow, proprietary, hard to share format and then acted surprised when it did not spread or survive.

What actually transfers

If you split your team's knowledge into the part that moves and the part that does not, the portable list is shorter than people expect and more useful than people expect. Agentic agents will take on these tasks, freeing people to focus on better analytics. But we live in the present, where teams are forced to use multiple query languages and tools still don’t talk to each other.

SQL is the obvious one. It is old, it is boring, it is everywhere, and every serious query engine you would put in front of a data lake speaks it. Learning SQL well is one of the highest return investments a security engineer can make, and it will still be paying off three platforms from now. One day this will not be an issue. 

Python is the second one, specifically for the kind of work that does not fit in a query. Pulling a case specific dataset into a notebook, joining it against asset and identity data, sessionizing it, and looking at it a few different ways. Analysts who can do that are not limited by what the SIEM UI is willing to show them.

Then there is the conceptual layer, which matters more than either language. Understanding schema and data modeling, which is why the OCSF work in post two pays off twice. Understanding aggregation, joins, time windows, and cardinality. Understanding enough statistics to know what a baseline is, what a rate looks like when it changes, and why counting distinct values is often more useful than matching a string. Knowing how to judge data quality, because timestamps, timezones, and truncated fields will wreck an investigation quietly and wreck a detection even more quietly.

Add version control and code review to that list. Detection logic and hunt queries are code. Teams that treat them as code, in git, reviewed by a second person, end up with a body of work that survives turnover and can be ported. Teams that keep everything in the platform UI end up with a body of work that evaporates on migration day.

None of that is tied to a vendor. All of it makes people better at the job they have right now, which is the part that makes it an easy sell internally.

The middle ground

Now the honest counterweight. You cannot run a SIEM with only portable skills, and anyone who tells you otherwise has not operated one.

Somebody on your team has to know the platform deeply. How ingest is configured and where it breaks. How the license or consumption model actually meters, because that is where surprise costs come from. How role based access is structured. Where the platform's own health telemetry lives. Which features are genuinely good and which ones are demo material. That knowledge is real, it is valuable, and it takes a year or more to build.

So the middle ground is not about avoiding vendor skills. It is about being deliberate about where they sit.

The rule I use is simple. Vendor specific skills are for operating the platform. Portable skills are for working with the data. Operating a platform is a specialty that a small number of people hold. Working with data is something every person on the team should be able to do.

When you frame it that way, a lot of decisions get easier. Sending two engineers to the vendor's deep administration training is a good use of money. Sending twelve analysts to a query language certification is money you will write off at the next migration. Building your hunt methodology around a feature that only exists in one product is a choice you will regret. Building it around a question you can ask of any dataset is not.

There is a useful test when someone is about to spend a week learning something. Ask whether the concept underneath transfers. Time windows, joins, aggregation, and baselining transfer. Macro syntax, app packaging, and dashboard XML do not. Learn the second category because you have to. Learn the first category because it compounds.

How to build skills that move with you

1. Split the role into platform operations and data work

Most job descriptions in this space are a mess. They ask for five years of one product and treat that as the qualification, which narrows your hiring pool to people whose expertise is, by definition, tied to a vendor.

Write two descriptions instead. Platform operations owns the SIEM itself, the pipeline configuration, the cost model, and the health of the system. Data work owns detection engineering, hunting, and analytics, and it is defined in terms of SQL, Python, data modeling, and security judgment rather than a product name.

This does two things. It widens your candidate pool considerably, because now you can hire people with a data background and teach them security context, which is very often faster than the reverse. And it makes clear internally that platform knowledge is a specialty a few people carry, not the price of admission for everyone else.

I have watched a team hire a data engineer with no SOC background into a detection engineering role. Six months in, that person was the one everyone went to for anything involving more than a few million events. The security context was learnable. The data instincts would have taken years to build.

2. Make the data lake where the analytics happen

This is the structural piece, and it is what makes everything else in this post possible.

Post two made the case for a data lake on cost and retention grounds. That case is real, but it undersells what you actually get. The lake is not just cheaper storage. It is the place where your team builds skills that are portable by construction, because the interfaces are open. Object storage, an open format like Parquet, and a query engine that speaks SQL. The skills people build there are not owned by anyone.

It is also the only place where a few of your most important programs can actually work properly.

Threat hunting needs depth and breadth. Real hunting means baselining authentication behavior across a year, stack counting parent and child process pairs across the whole estate, and asking whether you have ever seen an indicator further back than your SIEM's hot window allows. DFIR needs that same history plus room to work, because scoping an incident properly means going back as far as the intrusion goes rather than as far as your license allows, and then pulling that case data into a notebook to join against asset and identity records. IR gets the quiet version of the same benefit, where enrichment joins that are awkward and expensive inside a SIEM are ordinary table joins in a lake. Retro hunting an indicator across eighteen months should be a normal Tuesday, not a budget conversation. In the SIEM, every one of those questions carries a cost and a retention ceiling. In the lake, the difference is not a few percent. It is large enough to change which questions people are willing to ask, and that is the actual point.

And when someone eventually wants to do clustering, frequency analysis, or machine learning, the lake is where that work is possible. I am not going to oversell the machine learning part, because most teams get far more value from good aggregation than from a model. But the option only exists if the data is somewhere open and affordable.

Choose your query engine the same way post two said to choose your collection tooling. Ask whether the tool holds your data or just reads it. Athena, Trino, DuckDB, and Spark all read data you own in a format you control. The commercial platforms in this space can be fine too, as long as the data stays in open format in your storage and the engine is something you could swap. Store it open, and the engine becomes a decision you can revisit.

3. Treat data quality as detection engineering work

Detection quality is downstream of data quality, and it is not a close call. Most of the brittle detections I have seen were not written badly. They were written correctly against data that was inconsistent underneath them.

The pattern is always the same. The same event arrives from three sources under three different field names. One source sends timestamps in local time and nobody noticed. Command lines are truncated at a length that cuts off the interesting part. Host identifiers do not reconcile with the asset inventory, so the enrichment that was supposed to add context adds nothing for a good chunk of your estate.

Detections built on that will pass testing and fail in production, and they fail silently. A detection that does not fire does not page anyone. You find out months later, usually during an incident review, that the rule everyone trusted stopped matching anything when a parser changed.

Consistency matters more than richness here. A field that is always present and always means the same thing is worth more than a better field that shows up most of the time. That is the real argument for normalizing at the collection layer. Post two framed it as a portability play, and it is, but it is also a detection quality play, and that benefit shows up immediately instead of at the next migration.

This gets more important, not less, as detection moves toward behavioral methods and risk scoring. A string match either matches or it does not, so when the data changes underneath it you at least have a chance of noticing the rule went quiet. A baseline does not go quiet. It absorbs whatever you feed it and returns a confident number, so if parsing changed six months ago, part of what you are measuring is the parser change. Risk scoring is worse, because it aggregates signals across an entity over time and that only works if the entity is actually one entity. If a user shows up as three identities across your sources, the score splits three ways and never crosses the threshold. Entity resolution and consistent field semantics are not cleanup you get to the year after go live. They are the precondition for the whole approach working at all.

The control for all of this is backtesting, and it is a habit rather than a feature. You do not get to write a detection, watch it fire once, and walk away. Run it against history on a schedule and look at what comes back, not just whether it still returns rows but whether the volume looks like it used to and whether the entities still make sense. A rule that fired eight times a week for a year and has fired twice this month is telling you something, and it is usually about the data rather than the adversary. Backtest the data itself too. Track field presence, cardinality on the fields you depend on, event volume by source, and timestamp skew, because a field that used to be there nearly always and is now there most of the time is a broken integration nobody filed a ticket for. All of this is cheap in the lake and expensive in the SIEM, which is exactly why teams quietly stop doing it, and it is also how you learn a new detection's false positive rate before it goes live rather than after.

Garbage in, garbage out is a tired line. It is also the whole problem, and backtesting is how you find out which one you have.

People will point you at Sigma here, and it is worth knowing about. You write a rule in a vendor neutral format and compile it to whatever backend you are running. In practice it stays fairly niche. It handles log based detections and struggles with stateful correlation, sequencing, and risk aggregation, which is where a lot of your real value sits, and every backend still needs its own field mapping to work at all. The first post said content is never migrated, it is rebuilt, and that is still true. Detection engines are close to the definition of vendor specific and no rule format changes that. What you can control is what you feed them. Consistent, well named, well understood data turns the rebuild into a translation job instead of an archaeology project, and that is where the portability actually lives.

The skills point underneath all of this is that data quality belongs to detection engineering, not to whoever happens to own the pipeline. The people writing detections should be the ones who notice a field went missing, and they should have the access and the SQL to go check and the habit of checking without being asked. Post two suggested keeping a prioritized data quality backlog as part of dependency mapping. Give that backlog to the detection team. They feel the pain first, and they are the only ones who can tell you which twenty fields actually matter.

4. Teach the portable core on purpose

Skills do not develop because you bought licenses to a training platform. They develop because people do real work and get feedback on it.

Pick real questions. Put a couple of hours a week on the calendar and protect them, which is the hard part. Have people answer an actual question about your environment against the lake, in SQL, and then have someone else read the query. Not a formal review process, just a second set of eyes and a conversation about why the first approach was slow or wrong. That is where the learning happens.

Keep the output. Queries, notebooks, and detection logic all go in git with a short note about what question they answer. Over a year this turns into an internal library that is worth more than any certification, and it is a library that comes with you to the next platform.

Rotate people through platform operations so the deep vendor knowledge is held by more than one person, and so the rest of the team understands enough about the platform to work with it rather than around it.

Then use the migration itself. If you are already in one, it is the best training environment you will ever get, because everything is being rebuilt anyway and people are already off their normal patterns. Rebuilding a hunt in the lake instead of porting it query by query costs about the same amount of time and leaves you with something that survives the next migration.

5. Map your skills like you map your dependencies

Post two ended with dependency mapping. Do the same thing for people, and do it as part of the same yearly review so it does not become another orphaned initiative.

Write down who can do what. Who can operate the platform, who can write a good query, who can work in a notebook, who understands the schema, who knows how the pipeline is configured. Then find the single points of failure, because you will have them, and they are usually the people everyone likes working with most.

Every concentration is a risk with two faces. It is a resignation risk today and a migration risk later. You do not need to fix all of it. You need to know where it is and have a plan to spread the top few.

This is not a performance exercise and it should not be run like one. It is an architecture review that happens to be about people.

About the time this takes

I know how this reads to a team that is already behind. The first post in this series asked where the spare time was supposed to come from, and that question does not go away because the advice is good. So be clear about what is actually being proposed. This is not additional work stacked on top of the day job. It is the same work done in an order that pays you back. Documenting your sources now means finding them at your desk instead of at two in the morning in month four of a migration. One schema means fixing a parser once instead of chasing the same break through twelve detections. An hour a week of backtesting buys back the quarter where nobody noticed a rule had been dead since a parser change. Governance and preparation are not overhead, they are how you stop paying interest on work you skipped.

AI changes the math here more than most people have adjusted for. Models are genuinely good at the tedious end of data work: drafting a parser, mapping fields to a schema, translating a query from one language into another, writing the first version of the documentation your dependency map needs. They are not good at judgment. They cannot tell you which fields matter in your environment, whether an anomaly is worth waking someone up for, or what question to ask next. That split should sound familiar, because it is the same split this whole post is about. If a model can convert SPL to SQL in seconds, the value of memorizing syntax is falling and the value of knowing what to ask and whether the answer is right is not. Point your people at the second thing and let the tooling absorb the first.

This is a different way of working and it does not feel comfortable at the start. Teams used to living inside one product will find it slower for a while. It is still the right trade, because everything you build this way keeps working after the platform on the invoice changes.

What this actually buys you

Put post two and this post together and something changes about what a SIEM is to you.

Collection is decoupled, so sources are not tied to the platform. The schema is yours, so field names do not change when the platform does. The lake holds the depth, so retention is not a vendor conversation. Detection content still gets rebuilt, but against data you already understand, which makes it a translation job rather than a discovery one. And your team's analytical work happens in SQL and notebooks against data you own, so the skills are not tied to the platform either.

What is left for the SIEM to do is streaming detection, alerting, case management, and immediate response. That is a real job and a good product does it well. But it is a much smaller surface area than what most SIEMs occupy today, and a smaller surface area is a much smaller thing to learn.

That is what makes a SIEM close to plug and play. Not literally, because content still has to be rebuilt and no two platforms behave the same way. But when the thing you are replacing is a detection and response engine rather than the center of your entire security program, you are looking at a migration measured in quarters instead of years. Your team is not starting over. They are learning one new interface on top of skills they already have.

The other benefit shows up long before any migration. A team that works in data rather than in one product asks bigger questions because the questions are affordable, and answers a leadership question in an afternoon instead of a sprint. Observability went through this shift a few years ahead of security, and it is not a coincidence that the two leading vendors in that space, Datadog and Dynatrace, now sell on business outcomes rather than uptime dashboards. It started as collecting logs and metrics to keep systems running, which is a cost center conversation and gets funded like one. Then teams took the next step and used the same data to answer business questions. What does an outage actually cost. Which degradation loses customers and which one nobody notices. Where is capacity being paid for and not used. The data did not change much. What changed was the level at which it created value, and that changed how the function got funded.

Security is late to the same move, and that matters for funding more than most engineers expect. It is hard to get money for a data lake described as cheaper log storage, because that is a line item competing with other line items. It is much easier to get money for what the thing lets the business do. Incidents scoped in hours instead of days. Audit questions answered with a query rather than a project. Detection coverage you can prove rather than assert. A security function that is not hostage to one vendor's renewal letter. Same architecture, same spend, completely different conversation.

That leaves the third problem from the first post, and it is the one people are least comfortable talking about. You can own your data and build a team that can work anywhere, and still be stuck because of what somebody signed. Contracts, pricing models, and how to buy a SIEM without buying the trap along with it. That is next.


Next in the series: The Contract Problem. How to Buy a SIEM Without Signing Up for the Lock-In

1650575241014

Ed Bailey is a passionate engineering advocate with more than 20 years of experience in instrumenting a wide variety of applications, operating systems and hardware for operations and security observability. He has spent his career working to empower users with the ability to understand their technical environment and make the right data backed decisions quickly.

View all posts

Cribl, the AI Platform for Telemetry, empowers enterprises to manage and analyze telemetry for both humans and agents with no lock-in, no data loss, no compromises. Trusted by organizations worldwide, including half of the Fortune 100, Cribl gives customers the choice, control, and flexibility to build what’s next.

We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.

Get Started

Ready to SIEM a new way?

See how posture management, streaming detection, open retention, and AI-powered investigation change what your SOC can cover — and what that costs.