How To Take Prometheus Planet Scale: Massively Large Scale Metrics Installations
summary
Platform ingests ~41M/sec from 1.25M Prometheus endpoints with ~3 billion active series and ~8k queries/sec, up from 2M/sec in 2017. Built a Cortex‑inspired Prometheus stack with sharded TSDBs, Kubernetes operators and GitOps, added a field hint index (trigram/4‑gram fingerprints) to trim fan‑out, stream aggregators that drop labels and yield ~60% CPU and 80% memory savings, and implemented query push‑down and projection across leaf/zone/root tiers — prototype runs across Sydney, Amsterdam and US and improves ingest and query scale.
AI-written summary · about
chapters (14)
- 0:00Start
- 0:54Observability @ eBay
- 3:28Where it began...
- 6:18Challenges With Adopting Prometheus
- 7:24Initial Goals
- 10:42Storage Discovery & Management
- 11:19Challenges Still...
- 13:09Introducing Aggregators...
- 14:36Ingest Routing- Aggregation
- 15:15Let's Take Things Planet Scale...
- 21:00Deployment Architecture
- 25:30Query Routing
- 29:19Query Pushdown & Projection
- 39:08Questions?
