Prometheus Puzzlers: Traps, Pitfalls, and Corner Cases
summary
Hands‑on Prometheus puzzles showing concrete surprises: with 98 constant samples and six buckets a 99th percentile query estimates ~994ms because histogram_quantile linearly interpolates bucket counts (500ms and 900ms values collapse into the same bucket); a quantile in the top bucket returns the second‑highest bucket bound (1s); increase/rate with 15s scrape and counters every 60s can return anywhere between 0 and 3 due to extrapolation; single-sample counters evaluate as zero; adding request_id labels causes cardinality explosion and OOM.
AI-written summary · about
chapters (10)
- 0:00Prometheus and observability
- 3:42Observability essentials
- 9:18Prometheus as time-series DB
- 12:14Quantile illusion in histograms
- 19:51Highest histogram bucket quantiles
- 25:37Dangers of extrapolation in PromQL
- 31:07Alerting pitfalls with counters
- 35:29Missing metrics and alerts
- 39:11Cardinality trap and labels
- 42:29Summary and best practices
