Unlock High-Frequency Deployments without Blowing Up Prometheus
summary
Prometheus gets OOMs from Kubernetes rollouts because label changes and pod churn create thousands of stale time series that stay in memory until periodic compaction. The introduced stale-series compaction moves stale series from the head into blocks on disk, cutting peak memory by about 20–30% in Reddit’s experiments while sometimes raising CPU by up to 40% during compaction and rule evaluation. The feature is experimental with a known bug, config uses a stale series ratio trigger (example 0.4), and trade-offs depend on rule load and cluster size.
AI-written summary · about
chapters (11)
- 0:00High-frequency deployments with Prometheus
- 0:42Speaker background
- 3:06Stale time series causes
- 5:51Memory spikes from stale series
- 8:17Sharding Prometheus challenges
- 9:23Stale series compaction solution
- 10:27Configuring stale compaction
- 12:56CPU vs memory trade-offs
- 14:21Choosing stale series ratio
- 15:25Avoiding stale compaction
- 15:45Call to action
related talks
25mUnbreakable Prometheus: Leveraging Limits for Consistent Monitoring
35mConfiguring Prometheus for High Performance
18mArchitecting Growth: Scaling Tactics for Prometheus Metrics...
35mDoing Things Prometheus Can’t Do with Prometheus
34mAutoscale your Kubernetes Workload with Prometheus - Frederic Branczyk, CoreOS
42mPrometheus Intro and Deep Dive
03mPrometheus Project Updates
30mIntro: Prometheus