How We are Dealing with Metrics at Scale on GitLab.com
summary
Site-scale monitoring moves from 100k to 2.8M samples/sec, from 21 to 400 dashboards and to 2,600 recording rules; Prometheus is federated with Thanos and Thanos Ruler to avoid split-brain and to evaluate global SLOs. A json metrics catalog (stored in git, validated in CI) generates recording rules, multi-window multi-burn rate SLO alerts, and grafana dashboards via grafonnet. Alerts gain deep links into Grafana and other tools, partial_response is set to warn, and the system materially improves precision, recall and time-to-detect.
AI-written summary · about
chapters (11)
- 0:00Intro
- 4:03Key Metrics
- 6:07Alerts
- 8:20Metrics Catalog
- 14:16Simplest Approach
- 14:45Multiple Silos
- 15:36Split Brain Alerts
- 16:40Thanos Rule
- 17:50Example Configuration
- 19:00Conclusion
- 19:39Resources
related talks
37mCloud-native Metric Monitoring with Prometheus
50mMetrics and alerting with Prometheus and Grafana
31mEvolution of Metric Monitoring and Alerting: Upgrade Your Prometheus Today
14mFrom Metrics to Mastery: Monitoring vital signs of Kubernetes
31mHandling Billions of Metrics with Prometheus and Thanos
36mObservability at Scale With Monitoring as Code...
22mLook Ma, No Metrics: A Quest Towards Extracting SLOs
39mBeyond Logs: Achieving Observability with Selenium and Grafana