Cardinality Metrics for Monitoring and Observability
Modern cloud-native architectures and microservices have revolutionized IT environments, enabling unprecedented scalability and flexibility. However, these advancements have also led to a dramatic increase in the volume and cardinality of metrics data. This explosion in data complexity challenges the scalability and efficiency of observability tools, making it essential for organizations to understand and manage high cardinality metrics effectively.
In this glossary entry, we’ll explore what high cardinality means, what causes it, how it affects observability systems, and strategies to manage it effectively, particularly in modern cloud-native environments.
What is High Cardinality?
Cardinality refers to the number of unique values in a dataset. In the context of metrics, high cardinality indicates a large number of distinct time series or unique data points being tracked. For example, a metric tracking user IDs across millions of users would have high cardinality because each user ID represents a unique value.
High cardinality is particularly important in database systems and observability platforms because it directly impacts performance, storage requirements, and query efficiency. As the number of unique values grows, so do the computational demands for processing and analyzing the data. This makes managing high cardinality critical for modern monitoring systems that rely on real-time insights.
High cardinality can be both a blessing and a curse. On one hand, it provides granular insights into system behavior, enabling detailed analysis across multiple dimensions. On the other hand, it increases complexity, storage costs, and query latency. For organizations using time series databases or observability platforms, understanding high cardinality is essential to balance these tradeoffs effectively.
What Are The Causes of High Cardinality?
Cloud-native architectures have drastically increased metric cardinality compared to traditional monolithic systems. In legacy environments, metrics might be limited to thousands of time series. In contrast, Kubernetes-based microservices can generate hundreds of millions of unique metrics due to their distributed nature.
Cardinality reflects the number of unique combinations of metrics in your system. In modern cloud-native environments, cardinality can skyrocket, driving up storage costs and slowing down troubleshooting without effective observability strategies.
Here are five key causes of high cardinality in cloud-native systems:
Granular Telemetry Data
Modern observability tools often collect detailed telemetry data across multiple dimensions (e.g., user IDs, status codes). While this granularity enhances visibility, it also increases the total number of unique data points.
Dynamic Scaling
Container orchestration platforms like Kubernetes scale services up and down based on demand. Each instance generates its own set of metrics, leading to exponential growth in cardinality as instances scale up or down.
Distributed Architectures
Microservices architectures distribute functionality across numerous services and nodes. Each service generates its own metrics at various levels (e.g., per pod or per container), contributing to high cardinality.
Custom Tags and Labels
Many observability platforms allow users to add custom tags or labels to metrics (e.g., region, environment). While this enables more detailed filtering and querying, it also multiplies the number of unique time series.
Event-Driven Systems
Event-driven architectures generate metrics based on specific triggers (e.g., API calls or user interactions). Each event may produce unique metric combinations, further increasing cardinality.
These factors illustrate why high cardinality is a growing challenge for organizations adopting cloud-native technologies.
Low vs high cardinality in metrics
Cardinality exists on a spectrum. What qualifies as "high" depends on the environment and use case. For example:
Low Cardinality: Metrics with fewer unique values (e.g., HTTP status codes like 200 or 404). These are easier to store and query but offer limited dimensional insights.
High Cardinality: Metrics with many unique values (e.g., user IDs or IP addresses). These provide deeper insights but require more resources to manage.
The core tradeoff with high cardinality lies between dimensional insights and resource costs:
Pros: High-dimensional metrics enable granular analysis (e.g., pinpointing performance issues for specific users or regions).
Cons: Increased storage costs, query latency, and complexity can strain observability tools.
Organizations must evaluate whether the return on investment (ROI) from added dimensions justifies the cost of increased cardinality. Not all dimensions are equally valuable. Prioritizing high-signal, low-noise dimensions is essential for maintaining a scalable observability platform.
Solving the high cardinality problem in three steps
While granular telemetry data is appealing for its depth of insights, managing high cardinality requires strategic planning to avoid overwhelming observability systems. Here’s how organizations can address this challenge:
Step 1: Understand Your Data Landscape
The first step is gaining visibility into your data sources and identifying where high cardinality arises. Tools like Cribl Stream can help map out your telemetry pipeline by analyzing incoming data streams for excessive dimensional growth.
Key actions:
Audit existing metrics to find unnecessary or redundant dimensions.
Detect overuse of custom tags.
Group similar metrics together where possible.
Step 2: Implement Filtering & Aggregation
Not all data needs to be stored at full granularity. Filtering irrelevant metrics or aggregating them can significantly reduce cardinality without sacrificing key insights.
Best practices:
Filter out low-value dimensions (e.g., metrics from test environments).
Aggregate user-level metrics into higher-level summaries (e.g., per region or per role).
Use cardinality-aware routing to separate essential from non-essential data.
Step 3: Optimize Observability Infrastructure
Finally, ensure your observability tools are equipped to handle high-cardinality workloads efficiently. This includes leveraging technologies designed for scalability and performance optimization.
Recommendations:
Use time series databases optimized for cardinality-heavy workloads.
Leverage open source tools or commercial solutions like Cribl Stream to preprocess data.
Implement retention policies and sampling to limit unnecessary data growth.
How does Cribl control high cardinality & metrics growth?
Cribl offers powerful solutions for managing high-cardinality metrics within modern observability pipelines. By enabling organizations to filter, route, enrich, and aggregate telemetry data at scale, Cribl helps reduce unnecessary dimensional growth while preserving critical insights.
Key Features:
Data Filtering: Cribl Stream allows users to eliminate redundant or low-value metrics before they reach storage systems.
Aggregation: Combine similar data points into broader categories to reduce the overall metric count.
Scalability: Cribl’s architecture is designed for cloud-native environments where high-cardinality workloads are common.
Cribl is built for cloud-native environments and supports open source standards, making it an ideal solution for teams needing to control observability costs while maintaining visibility.
High-cardinality metrics are an inevitable challenge in modern IT environments driven by cloud-native architectures and microservices. However, with a clear understanding of its causes and impacts - and by leveraging solutions like Cribl - organizations can strike the right balance between granular insights and system efficiency.
