New to observability? Find out everything you need to know.
Cribl Stream is a vendor-agnostic observability pipeline that gives you the flexibility to collect, reduce, enrich, normalize, and route data from any source to any destination within your existing data infrastructure.
Learn More >Cribl Edge provides an intelligent, highly scalable edge-based data collection system for logs, metrics, and application data.
Learn More >Cribl Search turns the traditional search process on its head, allowing users to search data in place without having to collect/store first.
Learn More >The Cribl.Cloud platform gets you up and running fast without the hassle of running infrastructure.
Learn More >Cribl.Cloud Solution Brief
The fastest and easiest way to realize the value of an observability ecosystem.
Read Solution Brief >AppScope gives operators the visibility they need into application behavior, metrics and events with no configuration and no agent required.
Learn More >Explore Cribl’s Solutions by Integrations:
Explore Cribl’s Solutions by Industry:
Get this Gartner® report and learn why telemetry pipeline solutions represent a robust and largely untapped source of business insight beyond event and incident response.
Download Report >Observability Pipelines: Optimize Your Cloud with Exabeam and Cribl
It’s not about collecting ALL the data; it’s about collecting the RIGHT data.
Watch On-Demand >Try Your Own Cribl Sandbox
Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now >Get inspired by how our customers are innovating IT, security and observability. They inspire us daily!
Read Customer Stories >Sally Beauty Holdings
Sally Beauty Swaps LogStash and Syslog-ng with Cribl.Cloud for a Resilient Security and Observability Pipeline
Read Case Study >Experience a full version of Cribl Stream and Cribl Edge in the cloud.
Launch Now >Take Control of Your Observability Data with Cribl
Learn More >Cribl Corporate Overview
Cribl makes open observability a reality, giving you the freedom and flexibility to make choices instead of compromises.
Get the Guide >Stay up to date on all things Cribl and observability.
Visit the Newsroom >Cribl’s leadership team has built and launched category-defining products for some of the most innovative companies in the technology sector, and is supported by the world’s most elite investors.
Meet our Leaders >Join the Cribl herd! The smartest, funniest, most passionate goats you’ll ever meet.
Learn More >Want to learn more about Cribl from our sales experts? Send us your contact information and we’ll be in touch.
Talk to an Expert >During our March Cribl User Group livestream, Cribl’s own Eugene Katz covered some of the updates we made to our documentation on Architectural Considerations for deploying Cribl Stream.
Topics included our guidelines for determining the ideal number of worker nodes, accounting for throughput variability, and preparing for system failure. The full video has more information on these and other things to consider when determining the right balance between cost and risk for your organization.
When sizing your deployment, one of the first things to think about is keeping one worker available for maintenance purposes or in case another worker is down. Running at N-1 means there’s a backup available no matter what is happening with any particular machine or network.
Keeping that in mind, you can decide how many worker nodes are ideal for your particular setup. No matter which configuration you land on, remember that two virtual CPUs (vCPUs) per node are reserved by default for the API process and managing the system, as opposed to processing data.
You might be tempted to only have two worker nodes, with 30 worker processes each — but splitting up those worker processes over three or more worker nodes may actually give you more throughput per vCPU. Eugene walks us through the math here.
The general guidance for throughput is to have one physical core with 400 GB per day IN+OUT, or 200 GB per day for vCPUs. With this as our guide, if you set up your vCPU to handle a daily throughput of 51 GB per day, 51 GB IN + 51 GB OUT, you would have a total of 102 GB. This design is under the 200 GB recommendation for a virtual core.
But throughput is highly variable, so there are a few additional things to consider. The first is that oftentimes, Cribl Stream is used to write some portion of your data IN to an object store for full fidelity and then reduce and send copies of that same data to another destination. Routing to multiple destinations alters the otherwise equal ratio of data IN:OUT, giving you a total throughput greater than 102 GB.
Another thing to consider is spikes of data IN. Even a small spike can increase average throughput enough to require another vCPU.
Sources like Office 365 pull data periodically, leaving you with large data spikes that can easily be 5x that amount.
If everyone on your team has Windows events sitting on their laptops when they get to the office on Monday morning, your throughput graph will have a huge spike as soon as everyone logs on for the day.
Figuring out the actual behavior of any particular source is a challenge. You won’t have all these answers when planning your architecture, so you’ll have to make some best guesses. Once you start pulling the sources, some will start to average out — but keep one eye open for potential outliers.
Because of all the variables that come with managing all of your observability data, including the risk of DDoS and other cybersecurity attacks, at some point you’ll probably end up with a spike in data that exceeds the capacity of your system.
You’ll have to know your organization’s risk tolerance and decide ahead of time how high of a spike you’re expected to be able to support. If you want to lower your risk, you’ll need to throw more money at a solution — in the form of additional hardware or separating your worker groups.
An all-in-one worker group might be sufficient for smaller organizations and under certain conditions. Still, there are a lot of situations when multiple workers may be worth the additional investment.
Be sure to check out the full livestream for more information on when to consider multiple groups, how to better prepare your architecture to handle both expected and unexpected spikes in data, and how to keep your data flowing as smoothly as possible.
Experience a full version of Cribl Stream and Cribl Edge in the cloud with pre-made sources and destinations.