Alerting and Anomaly Detection – Best Friends Forever?
Talkat PrometheusDay Europe 2022 · on Tuesday, 2022-05-17· at 09:30· Pavilion 3, Room F | Level 2 | Central Forum + Live Stream, Feria Valencia, Spain + Live Stream·
summary
Argument: anomaly detection rarely works as primary pager logic because outages often look normal and most anomalies are harmless (World Cup traffic drops, 70% example). Prometheus uses SLO-based alerting and PromQL to build rule-based PromQL anomaly detection primarily for troubleshooting and informational alerts, not noisy pages. Machine learning belongs in automated troubleshooting systems that ingest logs, traces and metrics (Prometheus queried via API), produce ranked hypotheses, and help narrow causes — but not replace symptom-based, confidence-driven paging.
AI-written summary · about
chapters (6)
- 0:00Start
- 0:41Prometheus Intro and Deep Dive
- 7:04Symptom-Based Alerts versus Cause-Based Alerts
- 8:07Slo-Based Alerting
- 11:22Fraud Detection
- 27:05How To Avoid Noisy Alerting
related talks
30mPractical Anomaly Detection at Scale With PromQL
31mFirst Day Foresight: Anomaly Detection for Observability
38mWatch Everything, Watch Anything: Anomaly Detection
33mAlerting in Cloud Native Environments
25mSymptom-based Alerting for Machine Learning - What I Learned...
02mWhy you use Prometheus for monitoring?
34mML-Bound Anomaly Detection: Experiences Beyond Commodity Observability
19mThe Rise of AI in Anomaly Detection