June 7, 2022
If there was a question on if an enterprise needed an observability pipeline in 2019 or 2020, we now know the answer is: yes. The observability data management methods of the 2010s aren’t going to work in the 2020s. Data is growing too fast for us to ignore, and the need to gain intelligence from said data continues to grow in importance. Data (and access to it) is becoming a competitive edge for many enterprises today.
If you had to describe the 2010s in three words, it would be the mobile wave. During this decade, every interaction you had with a company likely had some sort of mobile interaction. From calling an Uber, streaming a movie, ordering batteries, signing a contract, or planning a trip with friends – every one of these interactions had a digital component, but likely also a mobile one. Everything was touted as being at your fingertips, but that also meant the number of data “streams” grew exponentially. As the mobile wave accelerated, the need for companies to analyze their observability data grew right with it. As the decade continued to pass, it became clear that organizations needed a strategy for managing observability data. This need is partly what led to the founding of Cribl. Our co-founders saw that enterprises were struggling to manage the data growth from a budget (both time AND money) and technology perspective. Even if a budget wasn’t a concern, the traditional path of shoving it into your analytics system wasn’t going to continue to scale. Searches became slow, so finding the information you needed became a challenge.
Turns out, we needed an observability pipeline. All data is not the same, so not all data should be treated the same. Some data should be stored in your analytics systems. Some should be sent to low-cost object storage. Some data needs to be masked (to protect PII) before it’s stored. Other types of data need to be enriched to provide additional context now and in the future.
The need for observability data that is smarter, better, faster, and stronger is clear. What might not always be clear for every organization, is the path to finding your solution.
Let’s consider another scenario. Let’s take someone who owns a landscaping company. Their expertise is maintaining lawns so the customers, and homeowners, enjoy a beautiful yard while limiting the time they spend doing it themselves. The owner of the business wants to spend their time making sure the employees have the best tools available to accomplish their goals for the customers. For them, lawnmowers, trimmers, hedge trimmers, work trucks, weed killer, fertilizer, and mulch are all tools they use to accomplish their goal. Would it make sense for the business owner to start manufacturing their own lawn mowers? What would you say to a friend who owns a landscaping company if they proposed that? Manufacturing a lawnmower might be a business they could get into, but it really has nothing to do with their core business.
Let’s pretend in a magical world, the tools needed to build lawnmowers were available. They’d have to acquire the resources, become experts in building them, not just using them, and then learn how to maintain them for the long run. From now on, they’ll be responsible for their lawnmowers and their customers. They won’t have the option to use a service center for repairs. There won’t be a community of people available to learn the best practices of getting maximum value from a readily available (and supported) tool. And if there’s unplanned downtime, they’ll have to invest in renting or buying lawnmowers to make sure their service doesn’t see degradation or worse yet negative sentiment. I don’t know about you, but it seems to me that the company should have focused on serving its customers and accomplishing its organizational goals instead of trying to build its own lawnmowers.
Let’s think about that same scenario again with observability pipelines. If your internal IT team decides to build their own observability pipeline, you’re now in the business of observability pipeline maintenance, support, and innovation. Are you prepared to handle all the possible data sources possible? Do you have the internal expertise to fix any bugs that come up? Do you have the expertise to understand how to build the tools to best enrich data from a source to a destination? Do you have the time to understand how fast the technology landscape is changing and the new sources entering the market? Are you able to deliver on the long terms expectations of your stakeholders? The list goes on and on.
Building your own observability pipeline technology is like building your own billing functionality or CRM tool. Technically, Kevin Garnett told us, anything is possible. Functionally, it makes more sense to leverage a solution purpose-built to solve your problems, so you can focus your team’s time and energy on only the things you can do to help your stakeholders serve your customers to the best of your organization’s ability.
Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure. We think it’s the best path forward as every enterprise in the world implements an observability pipeline. We’re so confident in that statement that we’re willing to let you process 1TB/day through our cloud service for free. But before trying out the full-featured cloud product, check out the free sandbox training courses that have sample data ready to use.