TELEMETRY PIPELINES ELEVATE THE HANDLING OF OPERATIONAL DATA, A COMPANION PIECE TO THE HYPE CYCLE, OUTLINES SIX BENEFITS AND FOUR RISKS FOR ENTERPRISES ADOPTING OBSERVABILITY PIPELINES. LET’S EXPLORE EACH IN TURN AND LEARN HOW CRIBL STREAM’S VENDOR AGNOSTIC APPROACH TO OBSERVABILITY DATA OFFERS THE BEST OPTION FOR TODAY’S TECHNOLOGY LEADERS.
GARTNER’S INNOVATION INSIGHT: TELEMETRY PIPELINES ELEVATE THE HANDLING OF OPERATIONAL DATA
With Gartner adding telemetry pipelines, also called observability pipelines, to the 2023 Hype Cycle for Monitoring & Observability, IT and Security leaders must understand not only how the technology benefits their enterprises, but also decipher and reduce the potential risks.
Benefits of Adopting Observability Pipelines
There is tremendous upside your team realizes with Cribl Stream as your observability pipeline, from cost reduction and data governance to faster response times. Here are six benefits highlighted by Gartner that our users realize daily.
1. Cost Reduction and Avoidance
The first benefit Gartner highlights, and one we hear often from our customers, is reducing the cost of ingesting observability data - the metrics, events, logs, and traces that are the backbone of today’s monitoring and cybersecurity practices. By removing redundant data, stripping whitespace, dropping unnecessary events, and even converting between formats, Cribl Stream users regularly see volume reductions ranging from 30-50%, depending on the data type and your needs.
This has two impacts. The first accrues immediately: you’ll be in a better position to manage the onslaught of data flooding into your security and monitoring tools. If you’re already bumping against ingest or budget thresholds, adding the control offered by an observability pipeline can quickly ease that pressure.
The second impact accrues at contract negotiation time. With more control over your observability data, future contract increases across your monitoring and security suite are greatly reduced. Cribl Stream users regularly cite forward looking cost avoidance as a key benefit.
2. Tackling Agent Sprawl
You know the story. After going through a POC for a new platform and deciding on a vendor, now you’re faced with installing the software agent across your fleet. With limited maintenance windows and configuration management processes, it can take nine months to a year to deploy agents across a large footprint. Observability pipelines like Cribl Stream make this easy.
By sitting between the sources and destinations of observability data, Cribl Stream allows you to take data from existing agents and reformat it for different destinations. Already have a fleet of Splunk Universal Forwarders installed? Use Cribl Stream to send that data to Exabeam, Dynatrace, Elastic, or any other destination, all in the right format.
3. Normalizing Tags and Identifiers
Tags and other identifiers are essential to efficient metrics collection and use, but managing that metadata is cumbersome. It might be spread across agents and their destinations. Evolving metadata over time is both error prone and expensive.
With Cribl Stream as your strategic control tier, you have a single place to add, update, and remove tags from all of your observability data in flight. Even better, with our leading GitOps support, all of your changes are tracked and auditable.
4. Lower Latency and Faster Response Time
Sometimes the event you get isn’t the event you want. It might have duplicate data, unnecessary payloads, or it might even be the wrong data type entirely. With Cribl Stream as your observability pipeline, you have a range of options for your data, such as:
Converting logs to metrics on the fly
Aggregating multiple events into a single event before forwarding
Suppressing duplicate events to save bandwidth and costs downstream
An observability pipeline puts you in control of your data.
5. Optimizing Observability Storage
When it comes to storing observability data, one size doesn’t fit all. A portion of your data may flow to a monitoring tool while another portion, formatted differently, routes to your SIEM of choice. On top of those, you’ll also need a copy of your data for compliance. At each step, you’re making choices about where data should be stored based on the value it provides.
Routing to different types of destinations, from security and monitoring platforms to object storage, is a core feature of observability pipelines. And once Cribl Stream has routed your data to an object store, you can replay it from that location, choosing the right data at the right time for everything from compliance audits to incident response scenarios.
6. Centralizing Data Governance
When it comes to governing data, observability data typically gets short shrift compared to transactional data, and for good reason. Transactional data is linked to a well understood business process, while observability is, in many ways, the exhaust of those business processes. This is misguided. Because of the volume and variety of observability data, it is often impossible to know when PII is sneaking into your logs.
An observability pipeline, like Cribl Stream, gives you a single place to govern and redact PII, from credit card numbers to passwords. You can also encrypt sensitive data in flight.
Managing Adoption Risks
No technology adoption cycle comes without risks. However, those risks can be better managed by selecting the right observability pipeline, backed by the right company. Below we explore some of the risks highlighted by Gartner, and how Cribl tackles them.
1. Tackling the Learning Curve
Getting staff productive on a new technology takes time, and each new product brings with it its own challenges. Your team may need to learn a language specific to that tool. Documentation and training offered by the vendor may be costly and sub-standard. These may sound trivial on day 1, but these hurdles introduce friction on day 2.
OBSERVABILITY PIPELINES CAN INTRODUCE A LEARNING CURVE TO OVERWORKED TEAMS. THIS IS WHY TRAINING AND CERTIFICATION AT CRIBL WILL ALWAYS BE FREE. CRIBL UNIVERSITY OFFERS MULTIPLE CERTIFICATIONS DEPENDING ON YOUR TEAM’S NEEDS.
Cribl differentiates itself here in two key ways. First, training and certification at Cribl will always be free. Cribl University offers multiple certifications depending on your team’s needs. Our free training makes it easy to ramp up and start getting value from Cribl Stream.
The second differentiator is how you interact with our observability pipeline. There’s no new language to learn or agent to deploy. We meet your team where their skills are today. With a little knowledge of basic Javascript and regular expressions, you can be productive with Cribl Stream in minutes, not months.
We want Cribl Stream to be easy for you to adopt and use. This is what you’d expect from the company that created the observability pipeline category and has the largest customer base actively telling us their needs.
2. Validating Results
Because an observability pipeline sits between your sources and destinations, there is a risk that bad data may make its way into your monitoring and security platforms. If you can’t preview your transformations, you could impact analysis or, worse, drive up costs as downstream systems struggle to process broken data.
Cribl Stream provides a unique feature to counter this risk. Stream offers a robust preview capability, allowing you to test your transformations over live data in real time. By previewing your work, you can be certain that what you see on the screen is what is delivered to your destinations. Coupled with our leading GitOps support, you can always roll back changes.
3. Managing the Observability Pipeline
When fully deployed, an observability pipeline is the core of your operational data. This makes it a critical piece of infrastructure. It must be managed and governed like any other essential component.
THERE ARE NO COST SAVINGS GUARANTEES WITH OBSERVABILITY PIPELINES. CRIBL STREAM OFFERS THE MOST EFFECTIVE ADMIN AND GOVERNANCE CAPABILITIES TO HELP YOUR COMPANY MANAGE ITS DATA MOST EFFECTIVELY.
For governance, Cribl Stream has an industry leading user experience for administrators to create role- and permissions-based projects comprising data sources, destinations, and processing. This ensures sensitive data isn’t accessible outside of the teams you designate. Likewise, it also guarantees that one team won’t interfere with the work of another team.
Cribl Stream also offers a rich API for management. Integrate it with your CI/CD tooling, or any other deployment or orchestration tooling for hands-off management.
4. It’s All About the Metrics
The final risk highlighted by Gartner is that there are no cost savings guarantees with observability pipelines. You have to understand your product’s pricing model to secure the cost benefits. Your chosen product must also provide fine-grained metrics over the data it processes.
For the first point, Cribl’s pricing model is straightforward. Whether running it yourself or taking advantage of Cribl.Cloud, the pricing is easy to understand and track.
Finally, Cribl Stream allows you to see exactly what’s happening to your data at each step. You can understand how much reduction, filtering, or enrichment is impacting your downstream data flows. Like our management capabilities, you can use the built-in dashboards or send your metrics to any destination of your choice.
Conclusion
Observability pipelines have gone from an obscure, niche product to being recognized by the world’s leading industry analyst firm. While there is no silver bullet for the observability challenges enterprises face today, Cribl Stream gives enterprises choice and control over their data. This allows you to take advantage of the benefits Gartner highlights while minimizing the potential for downside risk.
Check out our Buyer’s Guide for more detailed information. If you want to get started today, you can create a free account on Cribl.Cloud. Process up to one terabyte a day for free, forever
GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.
Disclaimer
Gartner does not endorse any vendor, product or service depicted in its research publications and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s Research & Advisory organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.