Demystifying Observability and Making it Work for You

This article is the final installment in a series that demystifies observability. The first three focused on the history of observability, dispelling myths around observability, and what observability is and what it can offer.

In this last article of the series (Check out part 1), I want to offer a complete definition of observability. There’s a common way of describing it — that observability allows you to determine what’s happening inside a system by watching what’s coming out of the system — but there are multiple parts associated with this, and observability is more of a concept or overarching architecture. In this post, I’ll share how we think about it here at Cribl, and in our webinar on September 7, we clarified things even further.

How Do You Collect the Enormous Amounts of Data Being Generated?

If you really think about it, your organization generates unlimited volumes of data. Your devices, systems, applications, hardware, and IoT devices are all generating some type of traffic that’s traversing your network whether they’re on or off premises. It’s interesting because there are times when you know what you’re monitoring and you collect it, but there are other times when you know what’s out there and you actually can’t afford to collect it. There are also times you might not even know what’s traversing across your environment.

All of this data is in different formats with different protocols – it could be unstructured or in a standardized or proprietary format. You run into the same problem as a fisherman going out with only a net and no bait or fishing rods; the problem is catching exactly what you’re looking for and what’s valuable to you.

Connect All of Your Current Systems of Analysis

Another to consider is your systems of analysis or storage. You have hundreds of offerings in this space and, if I’m being honest, they all have great value — if they didn’t, then people wouldn’t buy them. Depending on what specific needs you have, your organization will purchase and acquire a system that works right for them. Your ITOps, SecOps, DevOps, and AIOps teams will also have specific requirements for analyzing specific types of data in their own ways.

At Cribl, we assume you hired good people and that they know what they’re doing, so we’ll never try to tell you which one of these systems is best, or which one is going to meet your requirements. We also won’t tell you to rip out and replace any of your already existing infrastructure. Our view on observability is that we recognize there are a lot of systems out there, and what’s important is how you logically put them all together to gather the information you want — so what we provide instead are ways to complement the tools that you have already put in place.

How Well Do You Control, Observe, Visualize, and Operationalize Your Data?

These are the most important parts of observability — proper discovery, collection, and routing of data to the correct destination is the name of the game. Typically, you’ll have bespoke systems where one agent for an application is running on one machine and another agent for a different application is running on that same machine. We’ve even seen instances where you have five, ten, or more different agents all capturing data on the exact same machine. If this describes your system, then a lot of that data is redundant. You’re likely copying and sending it across the wire in parallel to the others. This isn’t always the case – an APM system might not capture exactly the same thing as a SEIM system – but the whole idea is that it’s forwarding the same stuff.

If you could send this data to a central hub, or what we now call a pipeline. then you can actually shape, reformat, optimize it, and then distribute it to 20, 30, or however many different destinations you want. This is where Cribl enters the conversation and observability really starts to happen. We give you the ability to send to each destination or system only what they need and process the data in a way that allows you to give your ITOps, SecOps, DevOps, and AIOps teams the data they want to see.

With Cribl Stream, you can reduce the amount of traffic you have and separate the wheat from the chaff so you don’t have to send hundreds, thousands, or even millions of null fields. You can aggregate data and avoid sending every event, log, metric, and trace to everyone or save and send all of them to object storage for retention purposes.

How We Define Observability

Observability isn’t a shiny new application, the next generation of monitoring, or an IT architectural revolution. And most importantly, it’s not complex. Observability is a collection of services, processes, and applications complementing and supporting each other in locating, collecting, storing, and analyzing enterprise data. It’s about having systems in place to collect all the data you need, shape it, and distribute it where it needs to go.

It’s also about recognizing that it’s going to continue to expand and putting open solutions in place to account for that. Make sure that any vendor you choose for an application, storage, or a data pipeline, you’ll be able to add new systems, protocols, and greater volumes of data along the way. You want to be able to see everything on your network today but you also need to prepare for the future.

Hope You Enjoyed Our Webinar on Observability

The swell of interest in observability — the technology, vendors, hype, marketing, and promises — has only spread confusion around observability and the benefits it can deliver. Which is why we hosted a webinar covering observability. If you didn’t catch it live, you can always watch the recording – full of straight answers and no marketing hype. We hope you left with a better understanding of observability and an appreciation for what it can do for you.

The fastest way to get started with Cribl Stream and Cribl Edge is to try the Free Cloud Sandboxes.