Monitoring the World: Scaling Thanos in Dynamic Prometheus Environments
summary
Cloudflare runs Thanos over nearly 500 Prometheus locations ingesting tens of billions of active series and ~50 PB for 13 months retention. The stack uses R2 and patched sidecars to allow automatic bucket creation per datacenter, time‑based sharding and per‑datacenter buckets, compactor jobs scheduled on edge machines during local night to use spare CPU, and a store sinker that chunks buckets into ~5 TB store units to meet SLOs. Query fan‑out is limited by a tiny label enforcer that rejects queries without external‑label filters.
AI-written summary · about
chapters (13)
- 0:00Thanos at Cloudflare
- 1:26Early Thanos setup
- 3:41Vertical scaling challenges
- 5:46Dynamic environment scaling
- 6:29SEPH to R2 migration
- 9:40Region bucket limitations
- 10:45Automating Thanos infra
- 12:32Compactors on Edge
- 14:41Centralizing stores
- 16:08Store sinker
- 17:32Query optimization
- 19:59Thanos components summary
- 21:05Q&A
related talks
26mPrometheus That Scales
54mThanos: Prometheus at Scale!
39mAbsorbing Thanos Infinite Powers for Multi-Cluster Telemetry
36mRunning the Observability As a Service For Your Teams With Thanos
15mScaling Observability Stack for AIOPS
31mHandling Billions of Metrics with Prometheus and Thanos
27mThanos: Highly Available, Pluggable, Long Term Metric Storage for Everyone!
27mHow Thanos Almost Snapped $100,000 from Our Infra Budget