Enterprises are accumulating more and more observability and security data in isolated silos, not much different than the dust and spare change under couches and chairs in your grandparent’s rarely-used living room. There is something of value in both examples, but the nature of the value is very unclear and hard to measure without a lot of effort.

Summary

• Data silos cost enterprises time and money with the need to master multiple search query languages to analyze data.
• Enterprises cannot afford to wait for manual data collection and analysis. It’s too time-consuming and your analysts will have to know multiple tool languages to run high-quality reports.
• Cribl Search bridges data silos with a single query language, to accelerate time to value and unlock the value of all observability and security data.

As I mentioned in my previous post regarding enterprise data silos, companies are collecting massive amounts of data at various disconnected locations. Popular silos are Elastic, Splunk, cloud object stores, and cheap file systems like NFS and EFS. Enterprises are breaking up their data primarily due to cost constraints imposed by conventional centralized logging models that have been pervasive for many years.

Traditional logging platforms, such as Elastic and Splunk, evolved over the last 10-15 years to consolidate all required data in one place to power analytics to support operations and security use cases. This model worked well for years until the crush data became too much and the costs of moving, storing and searching all this data became unsustainable. Enterprises tried to manage costs and complexity by:

1. Making choices about data to collect and not to collect
2. Add new analytics platforms that cost less to manage less important data
3. Store data in various formats and places based on cost and perceived risks

All of these choices create data silos that allow blind spots in coverage and limit how much value enterprises can get from data. Enterprises need help to bridge data silos and unlock the value from all of its observability and security data.

Searching for Value

The big question is how you get value from your data with it spread out everywhere. You are looking for spare change under the proverbial couch cushions and hoping amid the dust and lint to find the 5 bucks you need to buy a Popeye’s fried chicken sandwich for lunch. The enterprise search for data can be just as random. A common example:

Request: Business leaders want to know usage stats for every corporate website both internal and customer-facing for the past year.

Challenge: The relevant data is split between Splunk for customer facing sites, Elastic for internal sites and only at most 90 days of data is online. The rest of the data is on tape and in various cloud object stores. What is the best way to generate the requested report with the least effort?

Plan: Here is how I would address this request using only tools we have mentioned:

1. Make requests to the backup team to restore all the backed-up web server logs to temp directories for the Splunk and Elastic servers (7-14 business days)
2. Work with all the relevant application owners to find which object stores they used to back up web server logs, and move that data to the relevant Elastic and Splunk instances. (14-21 days)
3. Work with the Splunk and Elastic admins to restore the data to temp indexes. (5-7 days)
4. Work with the Splunk and Elastic searching experts to cleanse and then build reports. (2-3 days)

This is an incredibly common use case and also common to wait 28-45 business days for results.

Pulling the data from closed silos like enterprise tape arrays, and finding data that has been shelved to potentially dozens of object stores, is incredibly time-consuming. Each silo is a separate team with separate access and a separate ticket. Then you have to hunt through the backups looking for the right data.

Next, engage the Elastic and Splunk admins to restore the data to temp data stores. This can be a tricky process and takes some skill to get done without issues. Finally, engage the experts to run reports. This can be an obstacle since each analyst has to know the relevant tool language and know it well enough to produce high-quality reports.

This example only has 2 search languages to master, but it is more than common to use more. For example, many enterprises also use Amazon Athena to query S3 data stores. Athena uses SQL as its query language so it can bring even more complexity to the party of Elastic and Splunk query languages. SQL can mean more than just ‘Simple Query Language’.

Finally, this is way too long to get an answer for what should be a common report. Leaders can no longer wait on reporting to help drive business decisions. The underlying data needs to be available for reporting and not locked into dozens of disconnected silos.

The Value of Bridging Silos

What every enterprise needs is the ability to bridge data silos, ask complex questions, and get answers to the above use cases and much more.

This is why I am so excited about  Cribl Search. It offers 2 massive value opportunities.

First, bridging silos to unlock value from all enterprise observability and security data. Enterprises can achieve faster analytics with less effort because Cribl Search offers a common query language across multiple tools and silos. You don’t have to know Elastic, SQL, and Splunk query languages to get value across your data stores. This is everything! So many enterprises will find value in unexpected ways since they never had this capability before Cribl Search.

Second, Cribl Search does not require you to stop using your existing tools. Cribl Search will sit atop your existing tools and data to provide even more value. You will add new powerful capabilities without dealing with displacement cost issues that come up with other search tools.

Bottom Line

Cribl Search enables enterprises to unlock value from all of its observability and security data, with a common query language and rich analytics, by bridging data silos with a rich user experience. It also works with the tools you already have, so you do not have to stop using the tools you already have to get value from Cribl Search. You can get value from Day 1 instead of Month 12.

Try Cribl’s free, hosted Sandbox. I’d love to hear your feedback; after you run through the sandbox, connect with me on LinkedIn, or join our community Slack and let’s talk about your experience!