A reference architecture is a lovely document, but they rarely help engineers and architects implement their tools effectively. Most reference architectures offer plenty of suggestions and ideas, but not enough context. We will explore ways to make reference architectures more useful while reducing reliance on the vague and dreaded “It Depends.

Cribl has just released its first official reference architecture. The authors worked hard to give you the context and guardrails to make sure your Cribl deployment is successful right from the start. I’ll supplement their efforts, by sharing a mental model on how to approach your first Cribl Stream deployment.

Summary (TL;DR)

• Making a reference architecture useful by starting at the end
• Starting at the end gives you a goal to design towards
• Less ‘It depends” and more concrete requirements

Reference architectures, no matter how comprehensive they are, fall short of providing engineers and architects the necessary context they need to succeed. Diagrams and data points might not be sufficient to guide you in a unique security and observability data environment. So much of the wandering through the wilderness of the reference architecture has to do with the open-ended nature of the document. The document cannot anticipate every requirement, resulting in broad guidance that can be challenging to implement practically.

This gap between generic guidance and a useful deployment architecture is a problem in technology that hinders teams from getting fast value from their new toy. Unfortunately, this leaves the team in FAFO mode, which is not useful and requires rework that diminishes the initial value of the tool.

After many years of doing this wrong, and earning an advanced degree in FAFO along the way, I finally figured out that the best method for getting value from a reference architecture is to start at the end. This does not make sense at first glance but will make more sense as I give an example or two.

Start at the End to Get Started Quickly

The Cribl Reference Architecture gives you helpful guidance and formulas for deploying to production. But, deciding what to base your formulas on and what to design towards is a business conversation that cannot be entirely addressed in the reference architecture. So it’s crucial to follow some key steps and start at the end.

In the observability and security data world, it is common to base these formulas on daily data ingestion. This is a sound place to start since it is the baseline for minimum compute and storage requirements, but it’s more important to also consider a more critical factor related to your business needs. To address this, have a direct conversation with your business.

Your observability and security platforms are critical to monitoring your environment. It is how your teams see what is going on, so they can respond appropriately to issues and threats.

1. What happens if the platform goes offline due to a surge in data from your systems?
2. Do you know when/ at what thresholds your data systems will fail?
3. Are your observability and security platforms part of your business continuity plan?

Answering these critical business-level questions is necessary to size and architect your deployment correctly.

Example Scenario

Your team has opted for Cribl Stream to manage cost growth and improve data quality with the help of an advanced observability pipeline. As the data architect, you want to design the most optimal deployment architecture for your use cases. In addition to considering daily data volume, you also want to avoid the previous issues that occurred when a massive spike in data caused your infrastructure to fail.

Starting with the end in mind, you:

• Work with your business continuity team to designate your data infrastructure as critical.
• Ask the business at what point can your data infrastructure fail?
• Provide your business leaders with relevant data such as:
  • Our daily volume is 10 TB, but we have had 2 outages in the last 2 years caused by spikes to 3x-4x daily volume and 10 partial outages caused by spikes to 2x your daily volume.
• Offer options for how much it will cost to prevent the outages with the right deployment architecture.

Your leaders will weigh the risks with the cost of mitigation and give you an answer. Most likely the right solution will fall somewhere in between where partial outages are prevented and major outages become partial outages.

Your leaders have set the goal of sizing for 2x normal data volume, at 20 TB per day, which gives you a clear requirement to design towards using the Cribl Stream sizing guide. Asking the right questions helps you set a concrete goal for your design, rather than just guessing. To avoid the risk of being wrong, work with your leaders and define your end state prior to deploying your new Cribl Stream architecture.

My advice for anyone going through this process for the first time is to be prepared for the possibility that you might not like the decision your business leaders give you. You might feel like they are not listening to you. Remember their job is to base their decisions by weighing costs vs risk, and your job is to bring these risks to their attention. If your concerns are not addressed, it’s important to let it go and not let it consume you. Not every risk can be addressed, and that’s just a reality. However, make sure to document your concerns through emails and memos. This will come in handy when chaos ensues and you need to explain why a certain issue occurred. Then you get to wave the emails in the air when the same leaders want to know why your security monitoring failed during high load. It will make you feel better and maybe next time the leaders will listen to you.

Bottom Line

To speed up the deployment process, begin by defining the desired end state with your business leaders. Clear requirements are crucial for the successful deployment of any new tool. Ask your business leaders the right questions, and then work backward so you know how to implement the tool’s reference architecture to get the best results for your team.

I’d love to hear your feedback on getting started with Cribl Stream. Feedback is a gift, and I want to know if something doesn’t make sense or if I’m not covering something. Connect with me on LinkedIn or join our community Slack, and let’s talk about your experience deploying Cribl Stream.

The First 100 Days with Cribl Stream Series

• Part 1: Getting Started with Cribl Stream: Your First Hundred Days

Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.

We offer free training, certifications, and a generous free usage plan across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started. We also offer a hands-on Sandbox for those interested in how companies globally leverage our products for their data challenges.