November 7, 2022
Last week, on Halloween I hosted the Dawn of the Dead Data webinar – all about how Observability data growth is overwhelming our ability to handle it. Put another way – Our ability to collect data now exceeds our ability to analyze it. Thanks to all of you who attended, and I wanted to follow up with the attendees as well as those who were not able to join with a quick recap on what we discussed. If you missed the call or if you just want to watch it again, you can watch it on-demand here.
As promised, it wasn’t the same old marketing presentation on why one vendor’s solution was better than the next. Instead, we focused on the huge and growing volumes of collected data, which continues to grow at over 23% CAGR. This presents challenges in not just storing and paying for the storage, but also the ability to analyze the data, which was the whole purpose to collect it in the first place. And as I said in the first paragraph, our ability to collect data now exceeds our ability to analyze it. This isn’t just a system problem; it is a resource problem and the biggest of these license costs and limited budgets. As a result, collected data get saved off, maybe to be analyzed at a later time, but truth be told the vast amount of this data sits unexplored and unused. What we identified as Zombie data, is not quite dead, but not any value in its current state of just sitting in some data store, and potentially could impact the reliability, security, and performance of your network.
During the webinar we asked a few poll questions; the most interesting question was – Where is the most likely place Zombie data exists in your network? (Attendees could select more than one). Answers in order of preference are:
Answers 2-4 were mostly in-line with expectations, but #1 surprised me. The majority of responders believed the most likely place zombie data exists in their enterprise is in their systems of analysis. This tells me they spent the time and money to collect and ingest this data but have yet to truly examine it for the potential value and insights it could offer. It also implies that data collecting
After explaining what Zombie data is and how it is created, we then offered some ways to eliminate or at least reduce its volume, this included :
By the end of the webinar hopefully, we were able to provide you with the knowledge to examine your own enterprise and evaluate where zombie data may exist and some of the tools available to you to take action. Hopefully, you now have a better idea of how you can easily, and cost-effectively shape and reduce the data you collect to optimize the operation of your existing analysis system while saving a large amount money in both data ingestion and storage.