TL;DR on Data Silos

• Enterprises are generating vast amounts of observability and security data, but their current siloed search-based solutions limit the value from that data.
• The legacy model of forwarding all data to a central location drives up costs and lowers data utilization across teams.
• A new approach to search is required to unlock the full value of security and observability data and the use of data observability tools

Businesses are regularly experiencing 45% annual data growth, which is usually unsustainable given current data policies.In what way can this flood of data be cost-effectively managed while generating business value from critical data assets? Let’s break down data silos.

What are Data Silos?

History of Data Silos

For many good reasons, enterprises have long followed the practice of centralizing required and (hopefully) useful data into a single data observability platform, so they can use the data for query/alert/troubleshooting for operational and security use cases. Enterprises got a lot of value from this usage pattern.

This approach worked great for a while, but rapid data growth made this approach unsustainable. Data was growing not by gigs but by terabytes per day. The cost for license/storage/computing and the expertise to run it all exceeded the value of centralizing the data. Leaders started to notice the costs and pushed their teams to manage costs.

“Machine-generated data accounted for over 40% of internet data in 2020”

The Handbook on Research on Cloud Infrastructure for Big Data Analytics

To manage growth, enterprises tried a number of options to lower costs:

1. Ingest only the most important data into key platforms like its SIEM.
2. Use multiple data platforms in the same organization.
3. Store raw data on a file system or object storage.

Bits and pieces of data are literally sprawled all over the enterprise.

Only putting the most important data that goes into your SIEM reduces your ability to detect advanced threats, and slows down the incident response (IR) process due to a lack of ancillary data. By splitting up the enterprise using data on multiple platforms you create silos. This also slows down answering questions that cross business units and technology boundaries.

For instance, what if the enterprise security team wanted to know if application logs are catching hacking attempts? Finally, logging data to a file system or object may result in a checkbox indicating that sensitive data has been logged. Since there is no easy way to query this data, it will provide almost no business value.

Each of these options reduces short-term costs but limits how much value you can get from your observability and security data. Each option creates a silo that limits the scope and breadth of data to analyze. When you can’t tap into your observability and security data, you have a business problem at hand. This data is valuable and should be considered a business asset on par with all the other data in the enterprise.

The Business Value of Observability and Security Data

With some thought and creativity, this data can bring enormous value across the enterprise.

Let’s start with security:

1. Security observability, commonly called UEBA, uses behavioral data to identify potential threats. This is a valuable approach since it is harder to evade than static alerts, but this detection method requires a lot of clean consistent data to be effective. It is a costly technique to use with general purpose centralized data platforms.
2. It can be difficult for Incident Response (IR) teams to respond to incidents without all the possible data to answer. Questions such as “Was there lateral movement?, or “How was the endpoint initially breached?” can’t be easily answered. Retaining all the data to answer these questions in a traditional SIEM platform can be very expensive.

From the operations side:

1. This data can help marketers understand website usage and performance management.
2. It can help business operations do a better job with batch planning, by using ML to understand how resources should be allocated based on customer, time, and date.
3. Turn application logs into a new real-time source of data for business reporting, instead of relying on database replication which can take too long.

The options are endless, but require data to be effective. Which data platform you use does not matter, unless you have the right data to answer your questions.

The Value of Bridging Data Silos

What every enterprise needs is the ability to bridge data silos, to ask complex questions, and get answers to the above use cases and many more. Enterprises need to be able to bring all the data together and use rich analytics to visualize and share this data. It is a big challenge and a huge opportunity to unlock massive value from largely untouched data sources that have been long ignored.

One of the reasons why Cribl Search is exciting is because it offers 2 valuable opportunities. First, bridging silos to unlock value from all observability and security data. This is everything and so many enterprises will find value in unexpected ways since they never had this capability before Cribl Search.

The next big value is Cribl Search will not require you to stop using your existing tools. You can still drive value while using your existing tools. Cribl Search will sit atop your existing tools and data to provide even more value. You will add new powerful capabilities without dealing with displacement cost issues that come up with other search tools.

Bottom Line on Data Silos

Cribl Search enables enterprises to unlock value from all of its observability and security data, by bridging data silos with a rich user experience. It also works with the tools you already have, so you do not have to stop using the tools you already have to get value from Cribl Search. I cannot wait to start coming up with new ways to get value from observability and security data using Cribl Search.

Try Cribl’s free, hosted Stream Sandbox. I’d love to hear your feedback; after you run through the sandbox, connect with me on LinkedIn, or join our community Slack and let’s talk about your experience!