Can you, or will you? The real cost of running telemetry on a business data platform

Can you, or will you?

Snowflake and Databricks can handle almost any kind of data, including IT and security telemetry, if you put in enough effort. The big question is whether the effort is worth it: how long it takes to get useful results from your logs, whether your team has the skills or needs to hire or rent them, and what it costs, up front and every year after, compared to what the data is worth. For some teams, the answer is yes.

It's a different kind of data

These platforms were developed for business data, and both BI and AI rely on clean, structured, curated data. IT and security telemetry does not have this advantage. A sales table has the same columns every day. The format for Accounting and Finance’s income statement almost never changes and the inputs are stable because too many processes depend on stable data. People get fired for making mistakes with this data. 

Telemetry data does not have the luxury of stability. It is not considered a key business input and the accountability for mistakes is fairly limited. A firewall log can change the day the vendor ships a patch, or when an admin decides that a pipe-delimited event works better than a comma-delimited one. Process controls over telemetry data are limited, and this problem does not get nearly enough focus. 

I just worked with a very large enterprise that shows the gap. They run Databricks well for their business data, which is about 10 TB a day in three formats, which is a pretty simple environment. They want the same outcomes from their IT and security data, which is 500 TB a day in 800 formats. That's about fifty times the data in more than 260 times the formats, and the formats change or break often, whereas business formats hardly ever change. This is an enormously complex data estate for any commercial data lakehouse. 

Think of a data lakehouse as a kitchen set up for prepared ingredients. Telemetry shows up unprepped, in huge quantities, from hundreds of suppliers, and someone has to do the prep. You can do it inside the kitchen on metered time, or at the loading dock before anything goes into the walk-in.

What it costs

I'll use Databricks as the example. It organizes data in what's called the medallion architecture: bronze is the raw data, silver is the same data cleaned up and put into a consistent shape, and gold is the summarized data that dashboards and reports use. Each step between layers is a job that somebody has to write, schedule and maintain, and each job runs on compute that Databricks bills in DBUs, a unit of processing used per hour. Every transform hits that money tree. Snowflake bills compute in credits instead, but the pattern is the same.

At that enterprise, getting real analytics from even one IT or security data source takes months, and the major project milestones take up to a year. More broadly, getting material IT and security value from a commercial lakehouse can easily take a year, and in our experience the long-term cost doesn't come down afterward, while license costs keep going up.

We've also focused on adoption, because few IT and security users like using SQL, so prepared data can still go unused. Adoption is where the value shows up, and this is a classic mismatch between what users want and what the vendors offer. That may shift as the world moves to agentic AI, but we live in the present, and in the present the average IT and security user does not love SQL.

What worries me most is what happens when professional services goes away. New data sources keep arriving, so the work is never finished, and someone has to own handling 800+ formats and fix them every time one breaks. That break-fix time comes straight out of the time you meant to spend building better analytics.

Our math with medium to large customers shows that building and maintaining the medallion layers/transformation, including admin time, adds 15 to 20 percent to the total cost of the platform, mostly from the compute spent on transformation, both DBUs and the cloud hardware under them.

Two ways to fix it

None of this is an argument against Databricks or Snowflake for what they were built for. For enterprise BI data, they're the right choice. That data is structured and curated, its formats rarely change, and its value builds over time, because the same tables feed reporting, planning and models for years. The long-term value of that data set justifies the platform and the team it takes to run it. Telemetry is a different case. Much of it goes unused, and its value sits in the part you actually search and alert on, so the question is how much you want to pay to prepare all of it.

Option 1: Use a platform built for IT and security telemetry

You can use Cribl Stream and Edge to land your data in Cribl Search, which has its own Lakehouse Engines and Federated Search, along with Cribl Lake for low-cost storage. The difference is that many formats, high volume and constant change are what it was designed around. Your admins aren't mapping fields to tables by hand for every report, and the search experience is meant for the average IT and security user. That matters, because if users can't or won't use the tools, the value is lost. 

I recently worked with another company where 75% of its observability data was not searchable. They could not afford to put all of it into their primary observability platform. Using Cribl's tools, the team onboarded and gained visibility into the entire dataset in a few weeks. The data was messy, but they were able to expose datasets that had been unavailable for years. Incident resolution time was cut in half in the first six months. Now that the data is visible and usable, the team is starting a program to add agentic AI, which we expect to cut incident resolution time again by 30 to 50 percent. All of this saved them millions of dollars a year.

On a general-purpose platform, most of this team's time and money would have gone into getting hundreds of messy sources into a usable shape. Because the shaping was already handled, they put that time and money into visibility and faster incident resolution instead.

Option 2: Keep your lakehouse and fix the data first

If you're already committed to a commercial lakehouse, use Cribl Stream and Edge to normalize everything to one format before the data lands. Stream and Edge let a team with ordinary IT and security skills do the same shaping once, in flight, before anything is stored. So the two platforms differ in the cost of getting good data into place, not only in what the analytics can do.

I recommend OCSF, although any single stable format works, because the point is to limit how many formats you have to manage. Databricks then sees one format instead of hundreds, the transforms get much easier, and the Unity Catalog run is simple and routine. Stream is licensed on the data coming in, not the data going out, so sending normalized data on to Databricks, Snowflake, a low-cost archive or a SIEM adds no license cost. Hardware costs apply, and they depend on volume. You pay once to bring the data in and shape it, instead of paying each platform to do the same work. You also get to decide what to send. If you aren't using a data source in Databricks, there's no reason to route it there, and Databricks ends up working with a lot less data. That's my recommendation, not a measured number. Cribl Lake works better as low-cost storage than as a place to shape data, which is one more reason to do the shaping upstream.

The Bottom Line

Being able to do everything and actually doing it are different things. For IT and security telemetry, the gap shows up as months of work per source, constant break-fix, and a team that has to keep running it long after the project ends.

Before you commit, ask these questions:

• How many formats will we ingest, and how often do they change?

• What will transformation compute cost every year, not just in year one?

• Who owns break-fix after the consultants leave?

• Can our analysts use the result without writing SQL?

• How much of this data needs to land here at all?

That's where Cribl fits, and there are two ways to use it. You can land the data in Cribl's own Search and Lake, or you can keep your lakehouse and use Stream and Edge to send it in one format and a lot less data. Either way, the shaping happens once, before the data lands.

If you've priced out running telemetry on a general-purpose data platform, I'd like to hear how close your numbers came. Find me in the Cribl community Slack.