This IT Services Organization originally brought Cribl Stream into their company to help them with their data onboarding process. Stream simplified the consolidation of syslog-ng, some custom scripts, and other tools to make getting their data from source to destination easier.
“We refer to Cribl Stream as the conduit for our data — its pipelines keep everything flowing in the right direction.”
Cloud Solutions Senior Engineer
Implementing Stream had an immediate impact for the team — strategic event filtering led to real-world OpEx savings in their downstream SIEM tool integration and storage costs. But the impact of Stream went much further than that.
Easy Compliance with Evolving Federal Regulations
Back in 2021, the federal government issued an Executive Order (EO 14028) and subsequent Memorandum (M-21-31) to help improve the nation’s cybersecurity. It mandated federal agencies and government contractors to begin following specific guidelines for event log management.Using Stream, the company was able to assist their federal contracting teams with these new compliance requirements by collecting, routing, and delivering their in-scope customer logs to multiple agency SOCs.
“The use of Cribl Stream was an integral part of these accomplishments. It allowed us to collect and filter data from multiple sources, then route the results to each agency's secure destination in their preferred format and schema.”
Cloud Solutions Senior Engineer
Cutting Traffic Sent to Outsourced VSOC to Almost Zero
In his efforts to bring maximum value to his organization, the cloud solutions engineer has given some well-received demos over the years to his management team, mostly around technical use cases for Cribl. He’s had a lot of success so far, but he’s even more excited for his next demo that’s centered around cost savings.
He’s tested out the case for using Stream to filter the data sent to their Virtual Security Operations Center (VSOC). He’s sending what he calls “decision ready data” meaning they are only sending events that their security product is tuned to look to populate correlations and identify anomalies. In the past, the team has had to send the full logs, which causes the price per gigabyte costs to add up extremely quickly.
“By using Stream to filter the data that goes to our VSOC, we’ll end up with a 99.99% reduction in the amount of traffic we have to send. The cost savings are massive.”
Cloud Solutions Senior Engineer
Continued Visibility into VPC Flow Logs
Shortly after taking advantage of Cribl Stream, the cloud solutions engineer was notified of an organizational shift that would move their VPC flow logs from Cloudwatch into S3, where they would become unsearchable.
VPC flow logs are significant for operational troubleshooting and trend analysis — they can point to fundamental network issues and be used for trend analysis to spot potential issues, so having continued access to query them is important.
But from a cost perspective, sending them to Splunk didn’t make sense, so the team decided to bring on Cribl Search. They were able to take advantage of the cost savings and keep the ability to search their VPC flow logs in their new location.
The transition was pretty smooth:
“We set up the POV for Cribl Search before the cutover to S3. Everything worked out perfectly timing-wise — we had the implementation done on day one of the cutover and never lost the ability to search our data.”
Cloud Solutions Senior Engineer
With access to Cribl Search, they’ll also be able to transition NSG Flow Log storage from Splunk to S3, increasing ease of access and cutting more license and infrastructure costs along the way.
Using Cribl Search for Easy Troubleshooting
The cloud engineer has also had some personal wins since bringing Cribl Search into the fold. As an admin of multiple tools, it has helped him troubleshoot some longstanding issues, including a potential problem with a load balancer that needed a deep dive.
He knew that the company’s ELB logs were somewhere in S3, but they weren’t onboarded into Splunk, and he had no way to query them — until he remembered that he had Cribl Search in his toolkit. He pointed it to that S3 bucket and easily added a data source to be searched.
This is just one of many occasions where Cribl Search came in handy.
“There have been incidents where searching data was needed as soon as possible, but we weren’t always in a position to grab data and replay it without creating custom scripts or using up dev time. With Cribl Search, we now have immediate access to that data.”
Cloud Solutions Senior Engineer
Cold Tier of Storage for Infrequently Accessed Data
Cribl Search has also had a positive impact architecture-wise for the company and is driving significant changes to their processes. Search allows them to create a cold tier of storage in a data lake for infrequently accessed data across the company — it’s spread across multiple locations, but can still easily be queried through this centralized tool.
“Previously, the only option was to dump data into some long-term storage just to have it, knowing we’d never realistically ever search it. Cribl Search changes the game — now we can be confident in our ability to access any data when we need it.”
Cloud Solutions Senior Engineer
With Cribl Search, they can search and analyze the data in place, then determine what, if any, needs to be sent into their SIEM. This approach allows them to reload targeted data specific to an incident, rather than a bulk load of an entire time range, saving them both time and money.
Smooth Transition to Cloud Environment
The team has also fully transitioned from an on-premises Cribl implementation to a hybrid Cribl.Cloud environment to reduce their financial burden and the costs of managing their infrastructure. The cloud solutions engineer built the on-prem setup himself, using hybrid worker nodes kept in their own data centers.
“The flexibility with Cribl.Cloud’s consumption-based licensing is great, as opposed to feeling like you could be throwing away money if you don't use your exact daily license somehow. It was an easy transition for us, and we’re very happy with how it worked out.”
Cloud Solutions Senior Engineer
Best Practices for a DIY Cloud Migration
There’s no one-size-fits-all approach, but the best way to migrate to Cribl.Cloud in most cases would be to get all your ducks in a row and then just do a cutover. The cloud engineer did things a little differently in that he started moving individual data sources over one at a time — but he wanted to be sure everything worked as he cut over each piece of their architecture.
The cloud solutions engineer also tore down and rebuilt worker nodes instead of cloning the existing ones to help flesh out their process, which showed him how easy it was to deploy new ones.
“You don't really have to put too much thought into creating new worker nodes in Cribl Stream. You just deploy a server, run the script and it's done.”
Cloud Solutions Senior Engineer
Working with Kubernetes or EC2 could change things — but generally speaking, if you have more than a few worker nodes, putting them in a templatized format or using auto-scaling load balancers is a great approach. This way, you can just build it once and then run the scripts to install Cribl Stream. If you have an auto-scaling load balancer that’s set to make sure you always have your 20 or 30 worker nodes, it’ll spin them all up for you.
Using Cribl Stream and Cribl Search Together
Having Cribl Stream and Cribl Search has helped the company define its multi-tier data architecture. Useful data goes through Stream, and cold data is left in place while still being able to query it. Getting data out of Search and back through Stream is simple. Their scheduled searches aggregate large datasets and then export a summary result through Stream back to Splunk.
If they were trying to crunch the data in Splunk, they’d have to run long, exhaustive searches there, summarize the data, and then display it on a dashboard. Instead, they can just look at the data as they get it out of Cribl Search.
“It goes back to the cost of having all that data in Splunk — not just the ingest, but the disk space as well. The cost of ingesting the results from Cribl Search into Splunk is practically nothing. You could analyze a terabyte worth of data and then summarize it into a couple of kilobytes.”
Cloud Solutions Senior Engineer
In addition to the scheduled searches they perform every hour, the company also plans to have their team (and perhaps eventually, clients) perform ad hoc, templatized searches where they can plug in variables for IP, time, hostname, etc.
Looking To the Future
The cloud solutions engineer has even more plans for Cribl in the future. Moving their firewall logs would be a net positive for the organization which would incur some additional costs up front. But he’s confident management will understand the value, especially since they originally brought in Stream solely for source-to-destination routing and have gotten so much more out of it since.
When they first brought in Stream, they weren’t doing any reduction or transformation, but now they’re using Cribl functions to make data that was previously just being pushed straight through more usable and relevant.
He’s also trying to frame up using Cribl Stream to get metrics out of all of the company’s AWS accounts at scale. He has a vision for configuring organization-wide metric streams, where all their AWS accounts are consolidated, and all their metrics are in one location. All that data will be pushed through Stream, converted from JSON metrics into a Splunk metrics format, and then sent into Splunk to create dashboards. They’ll also do additional analysis on that data, looking for anomalous trend changes and other IoCs.
And while he is currently only using Cribl Search and Stream to discover, route and analyze data hosted in Amazon cloud storage, searching data hosted in Azure and Google clouds are on the horizon.
TL;DR
Real-world OpEx savings for SIEM tool integration and storage via strategic event filtering
Meet compliance requirements by collecting, routing, and delivering in-scope customer logs to multiple agency SOCs
Reduced traffic sent to outsourced VSOC to almost zero
Avoided downtime during cutover to S3
Point and search troubleshooting for admins
Added low-cost, cold-tier storage data lake
Seamless transition to Cribl.Cloud environment
Used synergistic effects of Stream and Search to define their multi-tier data model