Monitoring the monitor
summary
Shows a concrete tool, prommsd, that sends self-contained heartbeat alerts with annotations to tell where and when to alert so each Prometheus instance can be monitored without extra routing config. Uses Prometheus + Alertmanager + webhook receivers (PagerDuty/Healthchecks.io/DMS/Karma mentioned), demo kills Prometheus and prommsd detects loss after the evaluation/activation windows (30s threshold, 15s eval, heartbeats repeat ~5s) and forwards a no-connectivity alert. Notes a limitation: can't support dynamic alertmanager discovery. Source on GitHub and open source.
AI-written summary · about
chapters (8)
- 0:00Monitoring Prometheus
- 1:02Why Prometheus is complex
- 2:48Heartbeat alerts solution
- 4:58Prometheus SD intro
- 6:25Live Prometheus SD demo
- 9:41Testing PromSD with failure
- 11:06Architecture and use cases
- 13:19Summary and open source
related talks
41mMonitoring Like a Boss
50mMetrics and alerting with Prometheus and Grafana
29mParanoid Observability –Success Stories and Misfortunes observing an observability stack
02mWhy you use Prometheus for monitoring?
40mThe Prometheus monitoring system
23mAlerting in the Prometheus Ecosystem: The Past, Present and Future
31mEvolution of Metric Monitoring and Alerting: Upgrade Your Prometheus Today
17mMonitoring as Code