How enterprises store and split up observability and security data is a great analogy to how lint, spare change, and partially-eaten bags of popcorn end up under couch cushions. Or when you tell your kids to clean up the house when company is coming over and they stash their toys and your tools in various nooks and crannies.

The general approach to observability and security data is to opt for short-term tactical solutions to bigger issues. An engineer is told to lower storage costs so they dump data into object storage or cheap filesystems like NFS and this process is repeated all over the enterprise. Highly regulated data is managed according to standards, but what about everything else? Observability and security data have an enormous footprint, and enterprises are missing out on realizing its value by storing it all over the place. It’s kind of like all the little treasures strewn under our couch cushions and nooks of our homes.

Summary

• Observability and security data is stored everywhere
• Three typical use cases for how this data is stored and why
• The impact of data silos on the enterprise

As I mentioned in my previous post regarding enterprise data silos, data is spread out everywhere. You usually see three use cases:

1. Data left on endpoints to rotate out and disappear. Debug logging is a good example.
2. Data in different data platforms like Elastic and Splunk
3. Data in cheap storage, such as cloud object storage or NFS

Debug logging is a great example of high volume low value data. It is rare to see debug logging subject to retention requirements, it is rarely needed and the data is very large so enterprises avoid aggregating debug logs. Of course, when something is going wrong, developers need these logs to understand what is going wrong with their application.

Another typical use case is Security uses a SIEM like Exabeam, but the operations teams use Elastic for application logs under the assumption Elastic will be cheaper. Since the Security team does not need the application logs, then the general consensus is using another platform will not cause any issues.

Finally, it is very common to copy data that is not seen as valuable to cheap storage like cloud object storage or NFS. Some examples include web Server logs for corporate systems or firewall logs for dev and test regions. This type of data is commonly stored with the idea that it may be needed at some point, but there’s no attempt to get any value from it.

Analytics is how you get value from data and for most enterprises, but quality analytics is simply too expensive to apply too much of its enterprise observability and security data. Enterprises need better and cheaper options to manage, analyze and store their data. Data silos check the box that yes we have the data somewhere, but in the end it limits the value of the data beyond an audit checklist or narrow reporting that can be done per silo.

Bottom Line

With the coming release of Cribl Search, enterprises will have a complete vision to analyze and manage data where it lives. This will be a refreshing change from having to decide which data can be siloed and ignored, because traditional data processing platforms are too expensive. It also works with the tools you already have, so you do not have to stop using the tools you already have to get value from Cribl Search. You can get value from Day 1 instead of month 12.

What new ways will you come up with to get value from your observability and security when you implement Cribl Search? I can’t wait to see it!

Try Cribl’s free, hosted Stream Sandbox. I’d love to hear your feedback; after you run through the sandbox, connect with me on LinkedIn, or join our Community Slack and let’s talk about your experience!