SAAFE - A prioritized alerting model to troubleshoot your incidents
summary
Grafana Labs describes a working, open-source alerting pipeline that scores and ranks alerts by SAAFE categories (Saturation, Amend, Anomaly, Failure, Error) and severity to surface actionable signals first. It uses PromQL and Grafana recording rules to generate “assertions” (examples: CPU >80% for 10m, rollout detected via target_info subquery, error rate >5% → critical). A duration×weight score (info/warning=1, critical=2) ranks incidents, and an entity-map ties assertions to related services for causal timelines.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:01Anomaly detection alert problems
- 2:27Assertions for actionable alerts
- 4:14Incident response approach
- 6:44Prometheus alerts explained
- 7:45SAAFE alert taxonomy
- 8:48Saturation alerts examples
- 9:51Change detection rollouts
- 12:40Anomaly detection update
- 14:04Failure versus error alerts
- 15:48Ranking assertions by importance
- 17:54SAAFE open source resources
- 18:15SAAFE entity maps
- 20:51Q&A
related talks
37mSmart Data-driven Alerts (SDdA) and How They Make DevOps Life Easier
50mApplied Alerting Philosophy
32mWho Are Your Alerts For?
33mTackling Alert Fatigue with SLOs
30mPractical Anomaly Detection at Scale With PromQL
25mSymptom-based Alerting for Machine Learning - What I Learned...
40mNext-generation Alerting and Fault Detection
50mMetrics and alerting with Prometheus and Grafana