“WE MADE A CHANGE THAT SAVED US A SIGNIFICANT AMOUNT IN LICENSING COSTS IN 30 MINUTES.”
ENTERPRISE MONITORING AND OPERATIONS ARCHITECT, TRANSUNION
Ed Bailey and his colleagues in Enterprise Logging and Analytics at TransUnion are filling a tall order; with billions of logged events a day, the kind of scale they’re working with is not commonly found in the enterprise data pipeline world. Along with scale and volume, they manage a wide, dynamic array of data sources, each of which can represent significant effort to onboard and maintain for their internal customers. The team works with Cribl to meet the challenge head on.
Unsurprisingly, when you’re working at such a scale, efficiency is the word of the day — and with Cribl Stream, there are many ways to say it at TransUnion:
The mission-critical job of keeping TransUnion’s customers operating safely and smoothly means Ed’s team collaborates with a lot of internal customers who use data from the same source, but in a unique way. For example, one team may need DNS metrics sent to InfluxDB, and another team needs the full DNS logs, but sent to Splunk, while another team needs data in a third format. With Stream, each department gets what they need from the original data set, where they need it, without having to install additional agents or collectors.
Typically, when altering a data source’s format, you plan for some downtime while making the change, or at least a restart of agents and forwarders—but not at TransUnion:
Before deploying Stream, the TransUnion team often set up individual data streams to achieve the desired results, and then maintained those additional streams on an ongoing basis.
The volume of data TransUnion engages with on a daily basis is staggering. Bailey’s team uses Stream to help ensure what’s in that data is truly useful and valuable to the teams who work with it. Recently, they were able to massively reduce the scope of high-volume logging, such as DNS and Sysmon logs teams must examine from ~1TB a day to about 150GB – a near 20x reduction.
While simultaneously sending a full-fidelity set of the data to lower-cost, longer-term storage for potential future review, Stream checks each external request against a vetted list of known trusted data in real time, enabling more than half of the requests to be filtered out as uninteresting. It then looks up the remaining traffic against known bad thread lists, this time reducing the data volume down to 150GB and providing better data.
Windows Event Logs are notoriously bulky, but with Stream, Bailey’s team is able to strip out useless event codes and unnecessary fields, cutting the clutter and speeding the work of their internal customers.
AWS VPC Flow Logs are typically slow and expensive to deal with, but the TransUnion team runs them through Stream to suppress uninteresting or repetitive content and serves up the cleaned results in near-real time to the requesting department at a much-reduced cost.
The TransUnion team is excited to continue to leverage the many ways in which Cribl Stream adds to the efficiency of their streaming data operations.
Find out how your business can implement an observability pipeline to parse, restructure, and enrich data in flight, while cutting costs and simplifying operations.
Get Cribl, and take control of your data.
Cribl makes open observability a reality for today’s tech professionals. The Cribl product suite defies data gravity with radical levels of choice and control. Wherever the data comes from, wherever it needs to go, Cribl delivers the freedom and flexibility to make choices, not compromises. It’s enterprise software that doesn’t suck, enables tech professionals to do what they need to do, and gives them the ability to say “Yes.” With Cribl, companies have the power to control their data, get more out of existing investments, and shape the observability future. Founded in 2017, Cribl is a remote-first company with an office in San Francisco, CA. For more information, visit www.cribl.io or our LinkedIn, Twitter, or Slack community.