Paranoid Observability –Success Stories and Misfortunes observing an observability stack
summary
Runs Prometheus per cluster, Grafana per cluster, centralized Grafana + Mimir and a global Alertmanager; Loki and Tempo feed alerts too. Builds a meta instance Prometheus that scrapes other Prometheus and sends alerts, and a paired Alertmanager gossip setup across regions. Hits cardinality limits — local Prometheus reached ~75 million series vs the ~30 million guideline — so they replace full scrape backups with a smaller critical Prometheus that only opts in to essential metrics. Use an external Deadman's Nitch heartbeat; maintenance is two people.
AI-written summary · about
chapters (15)
- 0:00Introduction to talk
- 1:27Kubernetes observability basics
- 3:59Adding dashboards and alerts
- 5:25Scaling to global clusters
- 7:12Operations cluster setup
- 8:37Centralized alert management
- 9:37Adding logs and traces
- 10:40Introducing paranoia concepts
- 11:02Meta instance for scraping health
- 12:26Handling cluster outages
- 13:52Backup struggles and scaling limits
- 15:39Critical Prometheus for backups
- 17:26Deadman's Snitch for total failure
- 20:01Lessons learned and summary
- 20:43Q&A session
related talks
27mObservability with Prometheus and beyond
50mMetrics and alerting with Prometheus and Grafana
23mAlerting in the Prometheus Ecosystem: The Past, Present and Future
28mMonitoring Stack was yesterday – Observability Platform at scale!
24mPractical Kubernetes Monitoring with Prometheus
34mThe Grafana Stack: Building a Highly Available, Scalable Observability Platform
36mObservability at Scale With Monitoring as Code...
35mDoing Things Prometheus Can’t Do with Prometheus