Highlights
Successfully optimized data with Cribl Stream for predictable costs.
Doubled search performance at a flat cost via normalization.
Data routed to multiple destinations, reclaiming resources.
Cribl Stream enabled data enrichment and PII masking, boosting value and compliance.
Global Consumer Credit Reporting Company
A multinational data analytics and consumer credit reporting company, with its extensive data on over 1 billion people and businesses, faces the complex challenge of managing and processing data at an enormous scale. To avoid exponential increases in SIEM costs, the company embarked on a journey to develop and implement a robust data management strategy. This strategy is crucial for maintaining the efficiency and cost effectiveness of their services while still making security paramount important in their operations.
Many Splunk software customers are looking for ways to keep their costs flat or, at least, predictable because they can’t keep going back to the business every year asking to cover overage charges as a result of changing pricing models. Splunk Virtual Compute (SVC) was presented to the Global Consumer Credit Reporting Company as an affordable way to shift costs away from ingest-based pricing and make it attractive to migrate to the cloud. Through a combination of migration to SVC, best practices, and leveraging Cribl Stream they were able to provide better service to stakeholders and report predictable costs to the business.
Seeking partnership for success
As a trusted leader in the industry, the Global Consumer Credit Reporting Company is constantly processing billions of inputs from various vendors, agencies, consumers, and applications. The data being processed must be kept private and secure at all times, which requires a lot of infrastructure and tooling. The Platform Monitoring and Observability team delivers by keeping the data flowing and secure, even when the volume has grown over 1,000%, from 500GB to 10TB.
The Platform team was asked to migrate away from Splunk Cloud ingest-based pricing to SVC pricing. To make that transition successful, they needed to figure out how to optimize and increase the variety of data going into their Splunk Cloud environment, while keeping investment and resources flat. Since the Global Consumer Credit Reporting Company was already partnering with Cribl as the central hub for the data services they provide across their organization, they knew Cribl was the right partner to help them successfully transition to a new pricing model.
New architecture, who dis?
Increasing the types of data they were analyzing, while keeping costs flat was the primary goal, but as with any transformation, they wanted to take this opportunity to make multiple improvements, such as optimizing performance in analytics tools to run across broader data sets, reclaim resources for additional ingest, and route data to multiple environments based on business needs.
“Using Cribl Stream, we’ve optimized the data itself by removing null, redundant or non-relevant fields. This helps us to onboard more data sources while keeping our investment flat.”
DevOps Lead
To tackle their first objective – optimize performance – the Platform team used Cribl Stream pipelines to shape, normalize, and tune data before ingesting. To trim irrelevant data fields, the team uses Cribl Stream Functions like Eval, Lookup, Drop, or Suppress. They are also enforcing structured logging by checking an events’ structure or schema before delivering it to its destinations.
“We can take two dissimilar log sources and manipulate them in Cribl Stream so they look the same when they reach Splunk Cloud. We can use the same logic when searching across multiple datasets, so we return a more comprehensive result set, more efficiently.”
DevOps Lead
As part of this optimization, they were able to increase search performance by 2x–from running 750-800K searches per day pre-Cribl to running more than 2 million searches per day with Cribl–all at a flat cost, with the exception of additional storage to support the increased search volume.
“We can do more with the same infrastructure. We’re onboarding more data sources and doing twice as many searches with the same infrastructure.”
DevOps Lead
Routing to multiple destinations made easy
Next up they wanted to route data to multiple destinations, depending on the business needs of the data. In the case of compliance data, where retention periods may be up to seven years, you’re simply shifting the cost from ingesting all the data you need to store all of the data you might need access to. Reducing data ingest by leveraging an object storage, such as AWS S3, can reduce SVC costs by as much as 90%.
“Moving some data sets to AWS S3 minimizes the amount of storage we have to pay for.”
DevOps Lead
By sending compliance data to AWS S3 the Platform team was able to reclaim resources to onboard new data sources that were needed in the analytics systems. In addition, they also had the ability to send logs and metrics to other systems of analytics, such as Datadog. Using Cribl Stream to route the data, they are able to mirror Network Operations Center (NOC) dashboards in Datadog purely for redundancy purposes. When your business needs to be highly available, so do your monitoring tools.
Going beyond flattening costs to increasing value
Now in their second year, the Platform team’s spending has been flat thanks to enhanced architecture, tuning, and a focus on flattening their overall Splunk software footprint, thanks to Cribl. Another way they increase value from Cribl Stream, and optimize resource utilization, is by doing enrichment in Cribl Stream.
“Enrichment in Cribl is the way! I would much rather do all of the processing in Cribl Stream. And because lookups can evolve over time, enrichment at ingest time versus search time gives us more accurate data and saves time for analysts during an incident.”
DevOps Lead
The team is now using lookups for DNS, GeoIP, error code enrichment and asset tagging. It's common practice for them to leverage lookups from csv files, Redis, and keep them updated using a REST API.
In addition, the Platform team has used the masking capabilities to ensure PII wasn’t being sent to the analytics or storage solutions. While this was not an initial objective for the team, it helped ensure the privacy of the data and meet compliance standards.
“We did apply the Cribl masking function to our pipelines to make sure we aren’t getting any personally identifiable information (PII) where it shouldn’t be.”
DevOps Lead
The Platform Monitoring and Observability team now focuses heavily on improving the efficiency of data going into their destinations and the searches running in the analytics systems by having a dedicated engineer normalizing and enriching the data, enhancing the queries, and ensuring the right data is in the right tool. Cribl helps them control and shape what data goes where, giving them an easy path to provide monitoring redundancy and provide better service for their data consumers.
TL;DR
Sought to manage and process massive data volumes efficiently to avoid exponential increases in SIEM costs.
Successfully migrated to Splunk Virtual Compute (SVC) pricing, leveraging Cribl Stream to optimize data and ensure predictable costs.
Cribl Stream enabled the team to normalize, enrich, and optimize data, leading to a 2x increase in search performance (from 800K to over 2 million searches per day) at a flat cost.
Data is now routed to multiple destinations, including AWS S3 for compliance data, reclaiming resources and enabling new data sources.
Cribl Stream facilitates data enrichment and PII masking, further increasing value and meeting compliance standards.