Taming 50 Billion Time Series: Operating Global-Scale Prometheus Deployments on Kubernetes
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 12:00· Level 1 | Hall Entrance S10 | Room C·
summary
Claims operating 50 billion active time series across 20 regions using Cortex on Kubernetes and scaling each cell to ~1 billion active series, with per-cell S3-backed long-term storage and per-tenant isolation. Architecture uses cellular architecture + automated capacity controllers to pre-provision nodes, perform live cell migrations, and enforce quotas via shuffle sharding and token-bucket throttles. Deployments use zone-aware controllers for statefulset rollouts, deployment waves and canaries, and are moving to Carpenter for faster scaling; the system works in production with reduced blast radius and measurable resiliency.
AI-written summary · about
chapters (11)
- 0:00Introduction and challenges
- 1:02Evolution of the ecosystem
- 2:36Limitations of Prometheus
- 4:12Adopting Cortex
- 8:22Long-term storage and scale
- 9:54Cellular architecture
- 11:06Cellular design components
- 13:23Capacity management
- 14:27Deployment and testing
- 27:17Automation and Karpenter
- 28:22Future improvements
related talks
25mScaling Data Platforms: Managing Growth from Zero to Billions of Transactions
18mArchitecting Growth: Scaling Tactics for Prometheus Metrics...
26mScaling Multi-cluster Kubernetes at Teleport Cloud
34mPanel: Improving and Managing Kubernetes at Scale
32mMigrating 2 million CPU cores to Kubernetes
42m18 Months Before the Mast
37mManaging Large-Scale Kubernetes Clusters Effectively and Reliably
33mSpace Invaders for Kubernetes: Consolidating 50 Clusters at Scale