As an organization, you likely have many choices on where to store, analyze, and correlate your data. Those choices may change or iterate over time, so having an easy way to route data is needed. Enter Cribl Stream, which can route your data where it needs to go and save some effort, time, and money. It can help with organizational-wide initiatives like migrations and consolidations but can also help with smaller-scale initiatives and your day-to-day tasks of simply getting data in. Its flexibility allows you to get data where it needs to go faster and efficiently.

In this blog, we look at the options for building routes in Cribl Stream, the use cases around routing, and some potential benchmark savings or improvements. The benchmark data presented here has been derived from one or more existing Cribl customers and the benefits they have derived. Keep in mind your mileage may vary. Furthermore, Cribl Edge offers the exact routing mechanisms available in Cribl Stream so you can apply these use cases toward the source or edge of your environments.

Routing Overview

Cribl Sream is an observability pipeline that allows you to route data to one or more destinations. You can route high-value data to your systems of analysis while simultaneously sending data to low-cost storage for long-term retention. Furthermore, you can shape, transform, or reduce that data while it’s in flight. Not every system needs data in the same format, so you can quickly adjust and trim data as required for a given tool.

Within Cribl Stream, a route is simply a filter that determines where data needs to go. It comprises a Javascript filter, pipeline or pack for processing, and destination or output.

On the “Data Routes” page (shown above), you can create routes to filter, clone, cascade, and funnel data to packs and pipelines and order the routes as needed. With the Javascript filter, you can make your logic as custom as you need. For example, only pick a specific subnet of hosts from a given Cribl Stream Source for a given route. If all you need are connections between source and destinations with potentially a pipeline or pack in the middle, Cribl’s Quick Connect page is a visual option to the “Data Routes” option. See below for a screenshot of a Quick Connect view.

Now that we know how to build routes at a high level let’s discuss what potential benefits or ROI can be derived from a use case using Cribl Stream for routing.

Routing Potential Benefits & ROI

Use Case #1: Routing to Low-Cost Storage for Retention

With license costs for IT Analytics tools, SIEMs, and other tools increasing yearly, analysts are looking for ways to offload some of their long-term data into low-cost storage. Cloud providers offer cheap, low-cost object storage that typically fills this need and often is at least an order of magnitude cheaper than storing within your analysis system. Cribl Stream enables you to filter and route data to its appropriate destination, including low-cost storage wherever it may reside.

For example, you may send firewall logs to your SIEM and cloud object store. However, you can lower your retention period for that data in your SIEM to encompass the timeframe you need for most searches (typically 30 days). In this case, your object store will help retain your logs if you need them for compliance reasons or otherwise.

Benefits:

• Reduction of retention in your system of analysis
• Average of 10:1 cost savings in storage (depending on storage tiers & environment)
• Can leverage out of box life cycle management policies in popular cloud environments to further reduce and tier your long-term data

Use Case #2: Leverage Existing Collection Tier

Metric and log collection tiers are expensive, not only to administer but also to maintain. They can encompass hundreds if not thousands of agents (in larger environments), and that size and scale can be challenging to manage. Now, imagine that, on top of that, you need to maintain a collection tier for each analytics tool in your environment. It can get quite messy and fast!

Cribl Stream is designed to de-couple sources and destinations, breaking the one-to-one relationship between the collection agent and analytics tool. The message here is that you can leverage whatever you have today to send to multiple destinations with Cribl Stream in the middle, making any shaping or tagging changes needed for the various tools. Furthermore, without having the nth agent installed on your infrastructure, you should also be able to rest easy at night, knowing that there aren’t a handful of agents fighting for resources on your infrastructure.

Benefits:

• Reduction in collection tier infrastructure and maintenance costs
• Quick time to onboard data into other platforms (typically on the order of 45% reduction in the level of effort -LOE) and drive time to market
• Reduction in effort while migrating or evaluating new tools

Use Case #3: Speed up Getting Data In Processes

I am calling all administrators out there! What is your least favorite task that causes you nightmares at night? It’s probably the art of “getting data in” or GDI. As an administrator, GDI took up over 75% of my daily time. With all the different data formats and how dynamic data can be, trying to manage that was an art form.

Cribl Stream attempts to make this process easier. First off, everything can be done in the user interface. Setting up sources, capturing sample data live on the wire, and building appropriate pipelines for your data are all accomplished in the UI. The UI can be a “sandbox” to build and test your pipelines before deploying them to production. No more bouncing servers with each change you make or trying to decipher an outdated data sample provided to you. Furthermore, Cribl Stream enables you to move quicker when you already have onboarded sources that simply need to go to a new destination. With a quick route to the new destination, data administrators can quickly clone the data to a new destination, speeding up the process of GDI.

Benefits:

• Benchmark 45% reduction in effort for getting data in

Use Case #4: Routing data for Consolidations or Migrations

Migrations and consolidations can be rough! Trying to consolidate multiple environments or migrate to new environments with different sources and destinations can be time-consuming and high-risk. It typically requires coordinating sources and destinations during change windows and hot cutovers that could cause issues if something gets missed. Let’s not forget the nightmare rollback process if something were to go awry.

With Cribl Stream in place, migrations, and consolidations can be sped up while reducing risk. How, you might ask? Well, Stream allows you to route from sources to multiple destinations allowing you to route to multiple tools simultaneously that may be part of your consolidation or migration project. Once you have data routed to both, it allows you time to validate your data in your final destination before planning your cut-off.

Benefits:

• Speed up migrations by months (depending on size/scope of migration)
• Derisk migrations/consolidations
• Easily enables “warm migrations”

Use Case #5: Cross Domain Routing (Leveraging Compression)

Networks are limited! We can’t always fit as much data as we would like into our networks and at times, compression becomes our ally in trying to ship data. This can often be an issue at sites with remote or limited access, but can still be an issue if larger network links are already bogged down. Either way, Cribl Stream has a potential solution to this.

Because of Cribl’s ability to organize workers into worker groups, you have the ability to create and then daisy chain worker groups. By leveraging the Cribl Internal sources and destinations (Cribl TCP and Cribl HTTP), you can send data from one worker group to another across your domains. In doing so, you can leverage Gzip compression for data traversing from one worker group to another. This means you can now compress data from a given site to another site thus saving some network bandwidth in the process.

Benefits:

• Compression rates of 8:1 (average)
• Reduction of egress costs
• Reduced network load

Use Case #6: Tiering data for multiple Systems of Analysis

The legend of having one tool to solve an organization’s problems is just a myth. Trends show that organizations typically have dozens of tools, if not more, within their IT and security environments. However, not all tools are created equally- each will be designed to handle different data and purposes. Ultimately, having a data pipeline to front end these tools will make it easier to tier your data. Some data will be destined for pricier systems of analysis, while others might be destined for long-term storage on the odd chance you’ll ever need it. Cribl Stream allows you to tier your data and gain visibility into those data flows. While tiering data, customers typically achieve benchmark reduction levels of about 20-40% of data in their systems of analysis, allowing more space for data deemed critical to operations.

Benefits:

• Average licensing cost reduction average of 20-40% for systems of analysis (i.e., SIEMs, Monitoring tools), sometimes higher

Use Case #7: Enable Your Data Consumers to Self Service Their Data Feeds

Ever get stuck with trying to manage multiple data consumers in your environment? The security team may have one requirement for a specific data set, while the IT team may have another. And between all these stakeholders, they all want their data isolated from impacts caused by another team. Cribl Stream allows the creating of data feeds and subscriptions easily and faster than traditional methods. With Stream Projects, you can create isolated data spaces for your data consumers to manage their data, including routing and transforming their own data. Each team can then manage their own data flows without affecting another team’s data. Ultimately, this gives every team their own ability to engineer their data while still maintaining governance at a global data level.

Benefits:

• Enable data consumers to manage their data
• Speed up TTV (time to value) of your data – gain operational insights faster
• Federate data quicker in your organization
• Can save 100s+ of hours of effort across analytics/security admins across your organization

Use Case #8: Future Flexibility & Eliminating Vendor Lock-in

Business is all about making tough decisions. Needs and requirements change over the years, and what worked one day may not work the next. Having the flexibility to be agile in your tool selections is essential and can reduce risk and costs in the long term. Businesses move at a certain pace, and having data agility to keep up is critical. Vendor lock-in can sometimes hold organizations behind in implementing new strategies. Oftentimes, migrations become costly, complex, and time-bound. Cribl Stream gives you the ability to pivot to new tools and techniques as quickly as possible with the ability to create data routes to a variety of different tools. Furthermore, it allows you to transform that data into optimal form for each platform, allowing you to derive the most value from those platforms.

Benefits:

• FREEDOM!!!
• Speed up cloud migrations & vendor POC assessments
• Rehydrate systems of analysis with historical data, allowing for training of analytics and security models from day 1
• Align business tools with business strategies

Conclusion

In summary, Cribl Stream allows organizations to build, maintain, and manage routes to various destinations in their environments. In doing so, you can ensure the correct data is sent to the right destination in the correct form and, at times, offload that burden to allow your data consumers to self-serve. It will enable you to save costs while simultaneously taking down existing data silos in your organization to make you more data-agile.

Cribl, the Data Engine for IT and Security, empowers organizations to transform their data strategy. Customers use Cribl’s suite of products to collect, process, route, and analyze all IT and security data, delivering the flexibility, choice, and control required to adapt to their ever-changing needs.

We offer free training, certifications, and a free tier across our products. Our community Slack features Cribl engineers, partners, and customers who can answer your questions as you get started and continue to build and evolve. We also offer a variety of hands-on Sandboxes for those interested in how companies globally leverage our products for their data challenges.