The Sins of High Cardinality
summary
Prometheus metrics with free‑form labels like source/destination IPs and trace IDs cause explosive series growth — simple examples jump from 125 to 2,500 lines when adding protocol and direction, and cluster topology can reach 256 pods per node → exponential series with two‑to‑the‑16 orders of magnitude. Cilium/Hubble exposes these labels; ClickHouse or streaming aggregates handle long‑tail context better. Practical steps: enforce a cardinality budget, use tiered retention and recording/streaming aggregation rules, and drop or short‑retain metrics that are only diagnostic.
AI-written summary · about
chapters (6)
- 0:00High cardinality intro
- 3:14Why high cardinality burns resources
- 7:10Problem with exposing high cardinality
- 10:42Evaluating metric cardinality value
- 14:57Managing high cardinality strategies
- 18:08Recap and call to action
related talks
27mTaking Control of Metrics Growth and Cardinality: Tips for Maximizing Your Observability
20mWhat if Cardinality was free?
39mThe Great Cardinality Disasters of Our Time
56mUnwiring High Cardinality
18mArchitecting Growth: Scaling Tactics for Prometheus Metrics...
25mUnbreakable Prometheus: Leveraging Limits for Consistent Monitoring
13mLightning Talk: Taming Metric Cardinality: Practical Cost Reduction Wit...
19mCutting Through Metrics Cardinality Noise with VictoriaMetrics