High Available + Scalable Prometheus with Thanos in Alibaba
summary
Alibaba runs Prometheus at datacenter scale to monitor a cluster-management stack that handles 4 million TPS and ~10k RPS, and uses Thanos with a Thanos sidecar to get a global query plane instead of brittle federation. The deployment pairs local Prometheus retention with object storage, a custom target discovery component, and a central configuration/sync service; pain points include aggregation-rule maintenance, storage/replication overhead, query caching/consistency, and missing auth and fine-grained query safeguards.
AI-written summary · about
chapters (19)
- 0:00Intro
- 2:55Use Case
- 8:07Design Principles
- 11:12Agent Architecture
- 14:33Implementation
- 16:38Standing on the shoulders of giants
- 17:23Data Achievement
- 18:57Prometheus - Architecture overview
- 20:23Prometheus at Scale
- 21:30Prometheus Splitting By Use
- 22:03Prometheus Hierarchy
- 25:47Prometheus Horizontal Sharding
- 26:41Prometheus High Availability
- 28:14Hello, Thanos
- 28:57Global Querier
- 30:15Federated Querier
- 30:56Custom Target Discovery
- 32:03Configuration Management
- 32:59Enhancement . Authentication
related talks
10mUpstream-First, High Scale Prometheus Ecosystem
15mScaling Observability Stack for AIOPS
26mWhen Prometheus Can’t Take the Load Anymore
26mAgile Monitoring At Scale With Thanos
31mWhen Prometheus Can’t Take the Load Anymore
37mAutoscaling Multi-Cluster Observability with Thanos and Linkerd
02h 43mPrometheus from A to Z
32m“Gluing it all together” aka building a platform around Prometheus