“You might be a log analytics admin if …
You love the tech, but fear the next renewal.”
A customer recently shared a story demonstrating the value of choice and control over data. Their observability vendor finally provided a quote for an upcoming product renewal, which was three times the price of the current contract. With only eight weeks until renewal, they felt trapped with limited options. The customer had Cribl Stream deployed to manage security data, but the Cribl service owner recognized that it could also handle observability data. He colorfully expressed his opinion of the existing vendor and decided that a different approach was required. He convinced his leadership to dump the existing vendor and use Cribl to safely migrate their observability data to a new vendor.
The Cribl team switched to the new vendor in three weeks without losing data or compromising the customer’s monitoring posture. The customer saved money, rid themselves of a predatory vendor, and made a strong argument for a flexible, open data architecture. The Cribl service owner showed me the praise he received from senior leadership. The company had control of its data and acted accordingly. This story demonstrates the value of choice and control over your IT and security telemetry data. Cribl puts the customer in charge, rather than their vendors.
The root of the issue
Why do vendors behave this way, dropping unexpected, massive renewals on otherwise happy customers? The primary reason is that the vendor believes its customers have no other options. As a result, the customer must renew the contract, regardless of the price. Too many companies unintentionally put themselves in this position, and it is painful to experience.
I know this feeling well from earlier in my career. I had a large-scale logging and SIEM platform, and every decision we made was centered around what that SIEM vendor would support. When we struggled to afford increased data growth, we had to start making tough choices to drop data and build out a complex toolbox to contain growth, but the time invested in this data collection infrastructure consumed the engineering time we needed to get value from the platform. We were stuck between a rock and a hard place. You feel like you do not have any choices and have lost control to your vendor.
This all comes from the vendor-driven promise of “give me all your data, and I will solve all your problems.” The vendor provides the data collection and data processing layers. This is convenient in the short term since everything works together, but what happens when you need to evolve your data architecture to solve other problems, or worse, the same vendor increases cost by 3x? You are stuck. For example, your CISO buys a third-party UEBA solution that’s incompatible with your existing data platform, so you have no easy way to share data. Which means you are installing yet another agent to support the tool. Now you have data silos and duplicate data. It will only spiral as new requirements arise and you need to use your data in different ways.
Shift the focus to data
The key is focusing on your IT, security, and observability data strategy, rather than your tools. Ask yourself the following questions:
What data do I care about?
Where does it come from?
Who needs to use this data?
What is the data used for?
Where is my data retained?
This is where you begin to build your data strategy. Start with your data and then build your tool requirements from your data requirements. The driving principles behind your data strategy should be to decouple everything, match cost with value, and be as vendor-neutral as possible. This strategy enabled a three-week migration with no loss of monitoring posture. Follow these principles and you can unlock a wide range of value.
Start with decoupling your retention and build a data lake.
Retention is a good place to start since it is a significant barrier to tool and data portability. Any time retention is maintained in a vendor platform, the vendor owns your data. Your data is stuck in the tool, and you cannot easily get it out. Use a telemetry pipeline like Cribl Stream to route a full-fidelity copy of data to your data lake in a vendor-neutral format. This starts the process of democratizing data and opening access to the enterprise. Work with your corporate data team to plan out how your data is organized, but keep it simple to get started. Better to use basic object storage or something purpose built for IT and security data like Cribl Lake than go all in on something like a commercial data warehouse if you do not already know how to use it. Start with what you know and keep it simple and then evolve over time.
This is how you manage the age-old conundrum of having 10T of data and only 5T of license by decoupling retention and shifting to a tiered storage model that includes a data lake.
Match cost and value
A tiered storage model is where you identify data you know is needed versus data you might need and tier accordingly. Never manage retention in your high-cost o11y or security platform. Data that drives alerting or “right now” dashboards is routed to your primary platform, while data you might need, such as traces and debug logs, is routed to your data lake. Data classified as compliance is directly routed to object storage. Use your data lake to affordably “Log It All.” At the same time, you receive efficient and effective value from your costly primary platform, as it focuses on high-value use cases that use optimized datasets, rather than trying to store everything at a high price.
At a high level, your SIEM may cost $1 per GB, whereas even an expensive data lake can cost only $0.20 per GB. and object storage with an aggressive life-cycle policy can cost less than $0.01 cent per GB. A tiered storage model can conservatively enable storing 5x as much data for the same price.
With some governance, teams can promote and demote data as required so this does not have to be a static design. Pay close attention to who and how your data is used on your main platforms. If data is not being used at all or only being used in a specific time frame, consider demoting it to your data lake to help you match cost and value. If your needs change and a secondary dataset becomes critical, you can easily promote the dataset. This is an excellent example of how a telemetry pipeline can provide savings and flexibility by employing a strategy that matches cost and value.
Focus on open formats and flexibility
Decoupling your data collection from your data processing tiers enables you to freely share data between tools, as your data is in open formats and stored in standard platforms like AWS S3 or Azure Blob. It is very important to keep your formats and platforms as open as possible. Look for solutions that make it just as easy to extract data from the platform as it is to input data into the platform. This breaks the common "give me all your data lock-in" strategy. Follow this principle and you no longer have to consider what your data platform vendor supports when you make decisions about your overall data strategy. You can maintain retention in generic object storage in a vendor-neutral format. Now, your retention will be cheaper, and your data will be portable, which will reduce costs and boost the value of your data throughout the enterprise.
The bottom line
Build your IT and security data strategy to maximize the value of your data, control costs, and protect your enterprise from predatory vendors who are only looking out for their interests. Invest in Cribl’s data engine to grow the value of your IT and security data with the right approach and a strong foundation.
