3 Lessons About OpenTelemetry for Observability
Lesson 1 – Observability does not require OpenTelemetry
The most frequent question I field from customers and prospective customers alike about observability is the perceived dependency on OpenTelemetry. Cribl is a vendor-agnostic product suite to source, process, route, store, and analyze billions of events per second. As the data engine for IT and security, our customers and the Cribl Curious want to know about OpenTelemetry and the questions I hear follow a predictable pattern exposing a fundamental misunderstanding about the purpose of OpenTelemetry as an observability strategy. The conversation usually goes something like this:
Product Manager: “What does observability mean to you and your organization?”
O11y Observants: “We need to have OTLP Logs, Metrics, and Traces.”
PM: “Do you plan to use trace and span ID propagation to link logs and metrics to application traces?”
OO: “Yes!”
PM: “Fantastic, that is the key purpose for the OpenTelemetry semantics and standards.
Have you instrumented your applications to emit logs, metrics, and traces in OTLP format?
OO: <awkward silence>
OpenTelemetry (properly abbreviated as OTel) is primarily an application-centric approach to how data is structured as logs, metrics, traces, events, and resources. Standards are fantastic for data structures, transport mechanisms, and other key elements of interoperability. Ethernet, IP, NTP, and other standards are the unsung heroes of the internet age. The OpenTelemetry Signals are having a similar impact on observability, defining immutable standards for the format, content, and transportation of logs, metrics, traces, and the associated signals.
However, adopting OpenTelemetry is not a requirement in an observability strategy. If you can instrument an application to emit OpenTelemetry Traces, Metrics, and Logs, then adopting OpenTelemetry data standards as the preferred strategy makes sense. Without an application-first approach, OpenTelemetry can still provide a standardized data framework, especially if context propagation between metrics and logs is required. With OpenTelemetry, this concept is called a Resource and represents the entity, be it a server, host, pod, etc, that is generating the telemetry. These Resource Detectors are often the elements managed and maintained by IT and operations teams.
Adopting OpenTelemetry is a strong move towards a vendor-agnostic strategy for observability. However, it is not a perfect solution, or even the only solution, to build an observability strategy, especially if application tracing is not part of the equation. Logs and metrics power observability strategies for IT and Security teams with similar outcomes as tracing and metrics for application engineers.
Snippet from https://opentelemetry.io/docs/specs/otel/overview/#opentelemetry-client-architecture
Lesson 2 – OpenTelemetry does not require the OpenTelemetry Collector
As OpenTelemetry is primarily designed by and for an application-centric approach to telemetry, it makes perfect sense that the OpenTelemetry Collector is also very application developer/ DevOps/SRE-centric too. The Collector (abbreviated OTelCol) runs in two modes – as daemon alongside the application and as a Collector.
According to the OpenTelemetry Collector page, the collector “offers a vendor-agnostic implementation of how to receive, process and export telemetry data.” When a vendor wants to deploy their own version of the collector, it is still the OpenTelemetry Signals (logs, metrics, traces, etc.) that are vendor agnostic.
Cribl supports the ingress, processing, and egress of OpenTelemetry Signals (logs, metrics, and traces) even though it is not a deployment of the OpenTelemetry Collector. Where the OTelCol can be deployed in Agent Mode, uses Cribl Edge to collect OpenTelemetry signals, system metrics, and even system state too.
Agent Mode
From https://opentelemetry.io/docs/collector/deployment/agent/
Use Cribl Stream when centralized, petabyte-scale data processing is required. Stream Worker Nodes allow for horizontal scaling in a Worker Group with additional nodes and scaling vertically via Worker Processes by simply adding CPU capacity.
A Worker Group is similar to deploying the OpenTelemetry Collector in Gateway Mode using multiple instances of the OpenTelemetry Collector and the underlying OS. Where the OTelCol must be discreetly deployed and managed, a Stream Worker Group shares a managed configuration across all Workers and Worker Processes.
Gateway Mode
From https://opentelemetry.io/docs/collector/deployment/gateway/
Whether sending OpenTelemetry Signals directly from an instrumented application or forwarding them via OpenTelemetry Collector, Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale.
Lesson 3 – Observability is an outcome, not a feature
I start most of my introductions about my role as the Product Manager, Observability by joking that I am a PM without a product. I say that because you cannot buy observability and, as I quote in my blog Observability for Everyone, “Anyone who says differently is selling something.” Observability is about building a culture of well-instrumented technology, tools and mechanisms to acquire, process, store, visualize, and act on that data, and provide access to the data needed to ask and answer complex questions about business systems.
When a prospective customer starts a question with “Can you…?” followed by a technical explanation about their environment, it signals that they are struggling with one of those mechanisms essential to observability – telemetry acquisition, processing, storage, visualization, action (aka alerting/notification), and/or access. Achieving observability is rarely as simple as solving for a singular telemetry problem, it is about building repeatable patterns that meet current needs and mitigate future risks for accessing telemetry.
More and more Cribl customers are making the decision to expand the adoption of Cribl products – Edge, Stream, Search, and the recently announced, Lake – as the foundational architecture for powering their cultures of observability. Interoperability with Cloud, hardware, application, and other critical components allows those customers to focus on how to power their observability goals with data rather than revisiting data mechanisms for every business decision.
Why telemetry, not tooling, defines your observability culture
Nearly a decade ago, Torkel Ödegaard, the creator of Grafana and co-founder of Grafana Labs, wrote about the genesis of the phrase "Democratize Metrics," which shaped how I thought about monitoring and observability. OpenTelemetry has helped advance the idea Torkel and Raj Dutt, CEO and co-founder of Grafana Labs, proposed, expanding it to include logs and traces.
When data is well known, well structured, and accessible to the right people, in the right place, at the right time, that is a culture of observability. Observability is enabled by logs, metrics, and traces, and supported by the people your organization has assigned responsibility for availability, performance, and security.
That is the perspective Cribl brings to conversations about telemetry. Whether your teams have adopted OpenTelemetry or still rely on logs and metrics from legacy systems, the goal is not to force a single standard on every source. The goal is to give IT and Security teams a shared, open foundation where telemetry stays portable, interoperable, and usable, regardless of which standard, or none, a given application uses. Cribl Stream, Cribl Edge, Cribl Search, and Cribl Lake work together so your observability culture is not dictated by instrumentation choices you made years ago or vendor lock-in you did not choose. The data engine should adapt to how your organization actually generates telemetry, not the other way around. That is the choice, control, and flexibility we believe every team building a culture of observability deserves.
Lessons About OpenTelemetry for Observability FAQs
Does my organization need OpenTelemetry to achieve observability?
No. OpenTelemetry is one way to standardize telemetry, but logs and metrics can provide effective observability for IT and Security teams when application tracing isn't required.
Do I need to run the OpenTelemetry Collector to use OpenTelemetry Signals?
No. Cribl supports ingress, processing, and egress of OpenTelemetry Signals (logs, metrics, and traces) without deploying the OpenTelemetry Collector. Cribl Edge and Cribl Stream can handle this.
What is the difference between OpenTelemetry Collector Agent Mode and Gateway Mode?
Agent Mode runs the Collector as a daemon alongside the application, like Cribl Edge, which collects OpenTelemetry signals, system metrics, and system state at the source. Gateway Mode uses multiple Collector instances for centralized processing, like a Cribl Stream Worker Group that shares managed configuration across Workers and Worker Processes.
Why is observability considered an outcome rather than a feature?
Observability can't be purchased off the shelf. It is a culture built on well-instrumented technology and the mechanisms to acquire, process, store, visualize, and act on telemetry, backed by the people who use that data to answer complex questions about business systems.
Can Cribl convert Prometheus metrics into OTLP format?
Yes. The OTLP Metrics Function in Cribl Stream lets you take Prometheus metrics and translate them into OTLP Metric format without modifying the telemetry source.
