Agile Monitoring At Scale With Thanos
summary
MariaDB runs a Core Monitoring platform built on Prometheus, Thanos and Grafana that scales from PoC to 1000+ Kubernetes clusters, cloud VMs and on‑prem. It uses Thanos Receiver and sidecar ingestion, regional cores with sharded Thanos Store, 6‑hour local hot window and 2 years retention in object storage, plus a small global core that federates regional stores via Thanos Query and a Thanos Query Frontend. Autoscaling, Prometheus Adapter‑driven HPA on request rate, receiver filtering and a Slack‑centred alert format with one‑click silences reduce cost and alert fatigue.
AI-written summary · about
chapters (10)
- 0:00Introduction to observability
- 1:05Building a monitoring system
- 3:57System benefits and features
- 8:15Thanos implementation details
- 14:02Global cluster architecture
- 16:34Alerting and alert fatigue
- 18:43Cost efficiency and chaos engineering
- 21:17System evolution and restructuring
- 23:25Conclusion and SRE impact
- 24:31Q&A session
related talks
29mMonitoring multiple Kubernetes Clusters with Thanos
41mHigh Available + Scalable Prometheus with Thanos in Alibaba
30mIntro to Thanos: Scale Your Prometheus Monitoring With Ease
37mThanos: Prometheus in HA Mode with Persistent Storage
26mPrometheus That Scales
41mHigh Available + Scalable Prometheus with Thanos in Alibaba
10mUpstream-First, High Scale Prometheus Ecosystem
25mMulti-Cluster Observability: What you need to survive with a fleet