What is an Observability Engineer?

What is an observability engineer? Is it your SIEM admin? How about your application performance monitoring admin? Neither? Both? Observability engineering is more than administering a tool. There is more to it than data onboarding, writing parsers, and getting data in. As an observability tool admin, you work with data producers and consumers to get data in a human-readable and searchable format from the source to the analytics system. Too frequently, a tool admin knows only a fraction of the critical business use cases. Not knowing your data’s uses leads to challenges when asking for budget, headcount, and infrastructure. It leads to challenges when notifying internal stakeholders of outages, upgrades, and changes. It is difficult to inform critical customers, such as the SOC and NOC, of potential new gains and insights available from newly onboarded data.

What Does an Observability Engineer Do?

As an observability engineer, you are responsible for being a data steward and bridging the gap between the data producer and data consumer. You know data. You know how to discuss data to suggest and refine use cases and enrichment opportunities. You know how to decompose a use case into the component data sources and fields. This skill set takes time, practice, and talent. Recently, I had a chance to discuss this topic on a Cribl live stream that I’ve embedded below if you want to watch.

How can we empower observability tool admins to become observability engineers? Cribl Stream. Stream changes the types of conversations that you can have between producers, consumers, and the observability team. You now have a choice. You have the choice of what data needs to go to which system. You have the choice of which fields you need in your data for your multiple use cases. You have the choice of enrichment on the fly. You have the choice of data format.

The observability tool admin, now the observability engineer, can have a genuine conversation about the use case. No longer is the conversation simply, “what is the source?” “What is the destination?” “What retention do you require” “By when do you need it?”. Now, when the engineer has a conversation about the use case, they can talk about reducing unnecessary fields going to high-cost systems of analysis while still sending full-fidelity data to inexpensive archival storage. They can ease concerns when they show how easily data can be replayed from the archive back to the system of analysis when needed.

Now that the engineer is talking to data producers and consumers about use cases, they can document them. With well-documented use cases, a data dictionary starts to write itself. A data dictionary becomes a force multiplier for the organization. The c-suite knows the critical use cases. The SOC knows what is available (perhaps more importantly, what isn’t available) for threat hunting and detection engineering. All your stakeholders know what data is available, where it is, and available use cases.

The observability engineer now has the tool to become a responsible steward of the organization’s data. Moving from, “nope. we are going to hit our license cap.” to “yes! Let’s look at that data, see what’s valuable, and think about where you want to send it.”. The engineer can capture the reduction wins and show how their stewardship of the data has enabled many more use cases due to the choices they now have with the data.

The fastest way to get started with Cribl Stream and Cribl Edge is to try the Free Cloud Sandboxes.