If there was a question on if an enterprise needed an observability pipeline in 2019 or 2020, we now know the answer is: yes. The observability data management methods of the 2010s aren’t going to work in the 2020s. Data is growing too fast for us to ignore, and the need to gain intelligence from said data continues to grow in importance. Data (and access to it) is becoming a competitive edge for many enterprises today.

How Observability Data Was Managed in the 2010s

If you had to describe the 2010s in three words, it would be the mobile wave. During this decade, every interaction you had with a company likely had some sort of mobile interaction. From calling an Uber, streaming a movie, ordering batteries, signing a contract, or planning a trip with friends – every one of these interactions had a digital component, but likely also a mobile one. Everything was touted as being at your fingertips, but that also meant the number of data “streams” grew exponentially. As the mobile wave accelerated, the need for companies to analyze their observability data grew right with it. As the decade continued to pass, it became clear that organizations needed a strategy for managing observability data. This need is partly what led to the founding of Cribl. Our co-founders saw that enterprises were struggling to manage the data growth from a budget (both time AND money) and technology perspective. Even if a budget wasn’t a concern, the traditional path of shoving it into your analytics system wasn’t going to continue to scale. Searches became slow, so finding the information you needed became a challenge.

Enter Observability Pipelines

Turns out, we needed an observability pipeline. All data is not the same, so not all data should be treated the same. Some data should be stored in your analytics systems. Some should be sent to low-cost object storage. Some data needs to be masked (to protect PII) before it’s stored. Other types of data need to be enriched to provide additional context now and in the future.

The need for observability data that is smarter, better, faster, and stronger is clear. What might not always be clear for every organization, is the path to finding your solution.

Observability Pipelines: Build vs Buy?

Go Deeper on Build vs Buy in our On-Demand Webinar
The old saying, “there are multiple ways to skin a cat,” rings true for observability pipelines. When a team is convinced of the need, they’ll likely want to consider and investigate all their options. They’ll wonder: could we build this ourselves to save money and maintain full control? Yes, they might be able to. There are enough open-source solutions on the market that a team might be able to cobble together something to implement an observability pipeline. The question is, should they? Before building something, one should consider:

1. Can we afford to build it?
2. Can we afford to maintain it?
3. Do we have the expertise to build and maintain it, and/or how long will it take to learn?
4. Will (and when) we see a return on investment?
5. Is this project within the scope of our normal operations? (can we deliver at scale and on time for our stakeholders, and continue to do so in the future?)

Let’s consider another scenario. Let’s take someone who owns a landscaping company. Their expertise is maintaining lawns so the customers, and homeowners, enjoy a beautiful yard while limiting the time they spend doing it themselves. The owner of the business wants to spend their time making sure the employees have the best tools available to accomplish their goals for the customers. For them, lawnmowers, trimmers, hedge trimmers, work trucks, weed killer, fertilizer, and mulch are all tools they use to accomplish their goal. Would it make sense for the business owner to start manufacturing their own lawn mowers? What would you say to a friend who owns a landscaping company if they proposed that? Manufacturing a lawnmower might be a business they could get into, but it really has nothing to do with their core business.

Let’s pretend in a magical world, the tools needed to build lawnmowers were available. They’d have to acquire the resources, become experts in building them, not just using them, and then learn how to maintain them for the long run. From now on, they’ll be responsible for their lawnmowers and their customers. They won’t have the option to use a service center for repairs. There won’t be a community of people available to learn the best practices of getting maximum value from a readily available (and supported) tool. And if there’s unplanned downtime, they’ll have to invest in renting or buying lawnmowers to make sure their service doesn’t see degradation or worse yet negative sentiment. I don’t know about you, but it seems to me that the company should have focused on serving its customers and accomplishing its organizational goals instead of trying to build its own lawnmowers.

Let’s think about that same scenario again with observability pipelines. If your internal IT team decides to build their own observability pipeline, you’re now in the business of observability pipeline maintenance, support, and innovation. Are you prepared to handle all the possible data sources possible? Do you have the internal expertise to fix any bugs that come up? Do you have the expertise to understand how to build the tools to best enrich data from a source to a destination? Do you have the time to understand how fast the technology landscape is changing and the new sources entering the market? Are you able to deliver on the long terms expectations of your stakeholders? The list goes on and on.

Building your own observability pipeline technology is like building your own billing functionality or CRM tool. Technically, Kevin Garnett told us, anything is possible. Functionally, it makes more sense to leverage a solution purpose-built to solve your problems, so you can focus your team’s time and energy on only the things you can do to help your stakeholders serve your customers to the best of your organization’s ability.

Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure. We think it’s the best path forward as every enterprise in the world implements an observability pipeline. We’re so confident in that statement that we’re willing to let you process 1TB/day through our cloud service for free. But before trying out the full-featured cloud product, check out the free sandbox training courses that have sample data ready to use.

Webinar Q&A

How easily can I calculate how much Cribl.Cloud could potentially reduce my infrastructure costs?

We actually have an ROI Calculator tool that folks can use today at cribl.io. There are a few levers depending on your environment, and you can ultimately go through that calculation without having to interact with some of our sales reps if you don’t want to at this time.

Where can I see all the packs that are currently available?

All of our published packs are available in our Packs Dispensary. Since its introduction in April, the Packs Dispensary has seen over 800 individual downloads of packs and so they are benefiting our customers quite well.

What is the average daily amount of log data that was being processed by day by Cribl customers?.

The average Cribl customer processes 388 GB/day if we factor in that data growth is increasing at 23% CAGR, that will grow to 1.9 TB/day in approximately 5 years.

Go Deeper on Build vs Buy in our On-Demand Webinar