High Available + Scalable Prometheus with Thanos in Alibaba
summary
Alibaba runs Prometheus per datacenter for fine‑grained metrics at scale (supporting ~4 million TPS and 10K RPS) and uses Thanos sidecar + querier to provide a global, low‑latency view and long‑term retention; federation proves unreliable at this scale due to pull lag and aggregation maintenance. Central problems addressed: local 10s scrape intervals, huge replication and storage overhead, a custom k8s target discovery adapter, and a centralized Prometheus config manager to push rules and configs. Authentication, query caching and consistent time‑window reads remain work in progress.
AI-written summary · about
chapters (19)
- 0:00Start
- 2:55Use Case
- 8:07Design Principles
- 11:12Agent Architecture
- 14:33Implementation
- 16:38Standing on the shoulders of giants
- 17:23Data Achievement
- 18:57Prometheus - Architecture overview
- 20:23Prometheus at Scale
- 21:30Prometheus Splitting By Use
- 22:03Prometheus Hierarchy
- 25:47Prometheus Horizontal Sharding
- 26:41Prometheus High Availability
- 28:14Hello, Thanos
- 28:57Global Querier
- 30:15Federated Querier
- 30:56Custom Target Discovery
- 32:03Configuration Management
- 32:59Enhancement . Authentication
related talks
26mPrometheus That Scales
30mIntro to Thanos: Scale Your Prometheus Monitoring With Ease
37mThanos: Prometheus in HA Mode with Persistent Storage
29mMonitoring multiple Kubernetes Clusters with Thanos
26mAgile Monitoring At Scale With Thanos
21mThanos: Easier Than Ever to Scale Prometheus and Make It Highly Ava...
25mMulti-Cluster Observability: What you need to survive with a fleet
36mObservability with Prometheus and Beyond: Leveraging Cloud-native Technologies Outside of Cloud