Unbreakable Prometheus: Leveraging Limits for Consistent Monitoring
summary
A real-world plan for Prometheus on Kubernetes using kube-prometheus-stack and the Prometheus Operator: implement label limit 10→15 with soft/hard enforcement to stop explosive cardinality, set label name/value lengths 50/100 bytes to control series size, apply sample limit 250→500 per target and enforced target limits to predict memory (rough estimate ~37–38 MB per shaped target) and keep a server near 24 GB. Exceeded samples are dropped, prometheus logs and prometheus_target_scrape_pool_exceeded_label_limits_total increment, enabling alerts and autoscaling by shard.
AI-written summary · about
chapters (10)
- 0:00Introduction to monitoring limits
- 1:50Meet Unbounded Metrics
- 4:20Solving the label crisis
- 7:30Length label dilemma
- 11:00Simple limit challenge
- 13:50Target limit for stability
- 18:26Making Prometheus unbreakable
- 20:17Q&A on read path limits
- 21:41Space estimation for data shape
- 23:05Balancing limits and visibility
related talks
31mEvolving Prometheus for More Use Cases
18mArchitecting Growth: Scaling Tactics for Prometheus Metrics...
02h 43mPrometheus from A to Z
39mHow To Take Prometheus Planet Scale: Massively Large Scale Metrics Installations
35mPushing the Limits of Prometheus at Etsy
36mDoing Things Prometheus Can’t Do with Prometheus
35mDoing Things Prometheus Can’t Do with Prometheus
16mPrometheus 101