Designing for High-cardinality Metrics
summary
A stateless scrape-time reset algorithm converts cumulative counters into safe aggregates by resetting counters at flush and handling counter resets deterministically, implemented as ~8 lines of logic and deployed pre-ingest or inside Prometheus. Production reduces compute by 90% compute cost reduction, cuts a largest metric from 1M→27k series, and speeds heavy queries from ~20s to ~200ms (100x faster for some queries). Tradeoffs: block merges can duplicate samples (fixed by dropping the first duplicate), missing-sample ambiguity, and cases where counters never zigzag.
AI-written summary · about
chapters (10)
- 0:00High-cardinality metrics intro
- 1:06Pod churn pain
- 4:20Why databases struggle
- 6:31Scrape-time aggregation
- 8:17Cumulative counter issues
- 11:04Resetting counters safely
- 13:11Fixing aggregation errors
- 15:58Production results
- 17:46Trade-offs and limits
- 18:31Q&A
related talks
20mAggregating Metrics In-Flight: Challenges, Opportunities
40mCounting with Prometheus
38mAggregating Metrics In-Flight: Challenges and Opportunities
56mUnwiring High Cardinality
31mPrecision Over Proliferation: SRE Approach for Leaner, Smarter and Data-Driven..
08mStreamlining Telemetry Data
11mAggregating, Alerting and Graphing on Millions of Prometheus T...
19mA History of Automatic Aggregations