Handling Billions of Metrics with Prometheus and Thanos
summary
Intuit runs ~320 Kubernetes clusters producing about 2 billion active time series and shows why a single Prometheus HA pair fails as metrics grow from 300 to 4,000 metrics per pod. The stack uses Prometheus + Thanos sidecar, then centralizes rule evaluation in Thanos ruler to fix HPA aggregation, and separates pipelines with Thanos receive with 8 days retention for AIOps and short 6‑hour stores for Argo CD live metrics; a custom Prometheus Kafka adapter streams golden signals.
AI-written summary · about
chapters (18)
- 0:00Handling billions of metrics
- 1:06Metrics evolution at scale
- 2:10Early Prometheus scaling
- 3:56Golden signals and Kafka
- 4:39Scaling Prometheus with Thanos
- 6:04HPA issues with sharding
- 7:31AI Ops metrics use case
- 8:57Cost optimization with Thanos
- 11:26Argo Rollouts canary metrics
- 13:20Demo: Argo Rollouts metrics
- 15:51AI Ops anomaly detection
- 17:14AI Ops pipeline design
- 18:39Demo: Thanos receive for AI Ops
- 20:25Argo CD live metrics
- 21:49Custom Prometheus Kafka adapter
- 23:58Demo: Kafka adapter filtering
- 25:45Metrics add-ons overview
- 26:49Q&A: Prometheus sharding
related talks
15mScaling Observability Stack for AIOPS
26mPrometheus That Scales
25mMulti-Cluster Observability: What you need to survive with a fleet
10mUpstream-First, High Scale Prometheus Ecosystem
33mStory of Correlation: Integrating Thanos Metrics with Observabi...
22mMonitoring the World: Scaling Thanos in Dynamic Prometheus Environments
27mObservability with Prometheus and beyond
39mAbsorbing Thanos Infinite Powers for Multi-Cluster Telemetry