Cribl puts your IT and Security data at the center of your data management strategy and provides a one-stop shop for analyzing, collecting, processing, and routing it all at any scale. Try the Cribl suite of products and start building your data engine today!
Learn more ›Evolving demands placed on IT and Security teams are driving a new architecture for how observability data is captured, curated, and queried. This new architecture provides flexibility and control while managing the costs of increasing data volumes.
Read white paper ›Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn more ›Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn more ›Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn more ›Cribl Lake is a turnkey data lake solution that takes just minutes to get up and running — no data expertise needed. Leverage open formats, unified security with rich access controls, and central access to all IT and security data.
Learn more ›The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn more ›Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief ›Cribl Copilot gets your deployments up and running in minutes, not weeks or months.
Learn more ›AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn more ›Explore Cribl’s Solutions by Use Cases:
Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories ›Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study ›Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now ›Transform data management with Cribl, the Data Engine for IT and Security
Learn More ›Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide ›Stay up to date on all things Cribl and observability.
Visit the Newsroom ›Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders ›Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More ›Whether you’re just getting started or scaling up, the Cribl for Startups program gives you the tools and resources your company needs to be successful at every stage.
Learn More ›Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert ›Josh is a 25-year veteran of the tech industry who loves to talk about monitoring, observ... Read Moreability, OpenTelemetry, network telemetry, and all things nerdy. He has experience with Fortune 25 companies and pre-seed startups alike, across manufacturing, healthcare, government, and consulting verticals. Read Less
Did you know you can deploy Tetragon and parse high-volume logs with Cribl Edge? It’s true! Tetragon integrates seamlessly with Cribl Edge. This combination enhances monitoring capabilities in Linux environments. Have your cake and eat it, too.
With a combined Cribl and Isovalent solution, you can deliver deep insights into your workloads, optimizing for your specific operational requirements with zero loss of data fidelity. Tetragon allows you to extract the right data from the kernel and Cribl Edge filters, transform, and deliver the right data to the right place at the right time.
In this blog post, I will explain how to deploy Tetragon and Cribl Edge on a Linux instance and explore some of the telemetry that can be captured from the Linux kernel with Tetragon. The demo walkthrough sets up the Tetragon agent (the open-source version) and Cribl Edge to collect, transform, and route the Tetragon logs to a backend observability platform.
To begin, let’s answer, “What is eBPF?”
The eBPF documentation says it best:
“…eBPF is a revolutionary technology with origins in the Linux kernel that can run sandboxed programs in a privileged context such as the operating system kernel. It is used to safely and efficiently extend the kernel’s capabilities without requiring to change kernel source code or load kernel modules.”
Still a little confused? The diagram below from eBPF.io helps clear it up. eBPF operates at the kernel level and is leveraged by SDKs and a growing landscape of projects, like Tetragon and Cilium from Isovalent, to gain network, security, and observability insights.
You have probably heard about eBPF (or just BPF to the purists) in the context of monitoring for Kubernetes or the APM observability space. The pitch? See every call your application makes without making a single change to your code base. That is certainly an exciting proposition, even if you are not currently thinking about Kubernetes!
When Isovalent announced the 1.0 release of their Tetragon agent, including support for ARM processors, BPF became accessible to engineers outside of the Kubernetes and application spaces with a package-based install option.
For infrastructure and operations engineers, Tetragon can unlock an understanding of your Linux-based operating environments in exciting new ways and offers instrumentation policies for a myriad of use cases:
Tetragon provides detailed network monitoring and security insights, capturing logs about system calls, network connections, file access, system events, and more! Cribl Edge efficiently manages those logs as it filters, enriches, and routes the logs to a backend system for analysis and storage. Together, they offer a comprehensive view of system performance and security.
Tetragon’s challenge isn’t finding the right data, it’s finding space for all the right data. Data volumes quickly spike when you shift monitoring to the kernel from head or tail-based tracing, sampled metrics, periodic polling intervals, or even application-defined logging output.
In the observability space, the prevailing thought has been to collect all data in a central environment where it can be queried, summarized, and eventually aged out to make room for the new telemetry. Filtering at the source is always recommended, but without Fleet management agents across hundreds of instances (or hundreds of thousands!) quickly results in configuration chaos.
By adding Cribl Edge to the mix, parsing and filtering at the point of collection rather than a centralized database, optimizes telemetry flows, reduces storage costs, and improves query performance across the reduced data set.
If you are already using a log forwarder like Fluentbit, you might think, I’ll just add /var/log/tetragon/tetragon.log to my config and be done with it. That is an option, but here is what happens.
Tetragon logs are in JSON format but don’t include a message object, which many logging platforms usually require. Creating a message field by combining other fields in YAML and regex is often complex and sometimes unsupported. Defining and enforcing log standards, regardless of the source, shouldn’t be that hard.
Tetragon writes to the tetragon.log file by default. Each JSON object in the tetragon.log file varies in size but tends to be 2-3KB. On the test instance you will deploy in the demo, we measured 112 messages per minute without actual activity and no external applications. That is 0.32GB per day or 9.8GB per month per server for a lab server.
Let’s pause for a moment while you go check the price per GB to send logs to your preferred platform. Don’t worry. You don’t need to share your answer with us.
A verbose Tetragon profile or a misconfigured log forwarded could significantly increase logs being sent, increase costs, degrade performance, and add complexity when parsing the logs.
If you are wondering, “Should I even do eBPF-based monitoring? That log volume is a significant risk in our environment, and any unintended spike in log volume is going to be expensive”, the answer is “Yes, you can!”
Read on!
To fully understand each syscall, connection, file descriptor, and process, you need a solution tailored to answer detailed queries at any point in the data path. The Cribl product suite, built on the concept of a unified data engine engineered explicitly for IT and Security data, consists of Stream, Edge, and Search. Cribl offers complete control and flexibility to access, explore, discover, collect, and process data at scale as discreet products or as a holistic solution for observability telemetry.
Let’s look at each of the current offerings in more detail before moving on.
Cribl Edge, an agent for large-scale data collection, simplifies managing OS and application telemetry with centralized management that automatically detects and processes telemetry, deciding whether to send it to Cribl Stream or elsewhere. Why move complete debug logs to an expensive analytics store if they’re only useful temporarily? Edge allows users to directly interact with the endpoints and discover needed information before deciding whether to move it. In addition, Edge assists with agent consolidation efforts by providing robust centralized management and easily collecting and transforming data to send to a variety of destinations.
Cribl Stream is a robust streams processing engine focused on centralized parsing and processing of event data. Take any event source and use Stream to route, reduce, reformat, enrich, or otherwise structure data intended for any destination.
Cribl Search is an innovative new approach to finding and accessing data regardless of where it is landed and in any format. Search provides a federated solution built to separate the query engine from a storage medium. This delivers a unified query interface in a familiar and ergonomic pipe-delimited language that reaches into existing object stores filled with messy, unstructured, or structured datasets. It retrieves data without moving it or having to index it first. The same interface can also connect to APIs, databases, or existing tooling and combine results from all these disparate datasets.
“With great power comes great responsibility!”
– Stan Lee, Amazing Fantasy #15 (1962)
Okay, Stan probably wasn’t looking into the future to see the impact of eBPF as a new source of data in the observability and security space. Still, his words are no less prescient. eBPF unlocks a whole new world of visibility into Linux-based workloads (and soon Windows too!) and, with data forecasted to grow at 28% CAGR, according to the most recent IDC report – Worldwide IDC Global DataSphere Forecast, 2023-2027 – and adding kernel-centric observability telemetry is only going to exacerbate that growth.
We will build a solution that looks like the diagram below to help mitigate this data risk while still capturing valuable insights from the kernel. Tetragon will write observations into a log file on a Linux host, and Cribl Edge will read those log entries and deliver them to a data route. Here is where things start to get interesting!
In our case, the data route enables a single stream of telemetry where Tetragon logs will be sent to the appropriate data pipeline. In our case, we will install a pre-built Pack from the Cribl Dispensary that will accept data from a route and decide whether it should be sent to a pipeline for processing or dropped entirely. The pipeline is where the magic happens! We will take robust, nested JSON objects and extract, transform, and enrich them, including using lookups to translate obscure kernel nomenclature into a common, more readable format, and perform DNS resolutions for the source and destination IP addresses.
The transformed and enriched logs will then be returned to the original data route, where we will send it to an observability platform. Or, if you are like many organizations, send the logs to a data lake for long-term storage, a security platform, and an observability platform. As promised, you can have your cake and eat it too.
To set expectations, in our test environment, we achieved a 75% reduction in event length using the configurations in the demo walkthrough. In this limited sample set, we also saw a 79% reduction in total events. That’s a 94.8% reduction in total data ingest while normalizing Linux kernel event values to more human-readable values and performing DNS resolution on both source and destination addresses.
Ready to get started with Cribl Edge and Tetragon? Let’s GOAT!, as we say at Cribl. Check out the GitHub repo for a walkthrough on how to set it up.
Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.
We offer free training, certifications, and a generous free usage plan across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started. We also offer a hands-on Sandbox for those interested in how companies globally leverage our products for their data challenges.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.
Classic choice. Sadly, our website is designed for all modern supported browsers like Edge, Chrome, Firefox, and Safari
Got one of those handy?