Thanos: Prometheus in HA Mode with Persistent Storage
summary
Describes replacing an expensive enterprise monitoring stack by deploying Prometheus + Thanos on k3s (Rancher-managed) with Grafana, using external_labels per cluster so Thanos Querier centralizes metrics from many Prometheus instances. Uses S3 object storage with a 15-day S3 lifecycle and a Thanos Compactor to remove redundancies and reduce storage costs; deployment and onboarding (~30–35 people) use Flux CD and exporters, yielding a horizontally scalable platform that cuts annual costs to about $8,000.
AI-written summary · about
chapters (10)
- 0:00Thanos monitoring
- 7:46Enterprise monitoring issues
- 10:14K8s challenges for Prometheus
- 13:28Thanos storage scalability
- 17:14Architecture demo
- 22:14Query setup Prometheus & Thanos
- 26:34Adding clusters to monitoring
- 27:18Next steps: Loki logs
- 31:02Cost savings results
- 34:34Kilo VPN details
related talks
30mIntro to Thanos: Scale Your Prometheus Monitoring With Ease
26mPrometheus That Scales
29mMonitoring multiple Kubernetes Clusters with Thanos
41mHigh Available + Scalable Prometheus with Thanos in Alibaba
25mMulti-Cluster Observability: What you need to survive with a fleet
21mThanos: Easier Than Ever to Scale Prometheus and Make It Highly Ava...
26mAgile Monitoring At Scale With Thanos
31mObservability: Dust Off Prometheus with VictoriaMetrics