Symptom-based Alerting for Machine Learning - What I Learned...
summary
More than 30 ML use cases and roughly 30 production models show that silent failures (for example an input unit change from seconds to milliseconds) break quality without backend alerts and can swamp gains (teams gain ~4% from improvements but lose ~15% from unnoticed regressions). Priority is outputs first: monitor postprocessed predictions, production evaluation when ground truth exists, and stakeholder “fear” signals. Root cause uses input/training-data checks and distribution tests (KS/PSI). Implementation often uses Prometheus/Grafana, Prometheus histograms, S3 logging and a batch job every 3–10 minutes, with careful ramping to avoid alert fatigue.
AI-written summary · about
chapters (8)
- 0:00Introduction to ML monitoring
- 1:39Backend basics and gaps
- 3:37Silent failures in ML
- 6:41Prioritizing output signals
- 8:37Evaluating model performance
- 13:27Monitoring service outputs
- 17:04Input monitoring strategies
- 19:17Tooling and implementation
related talks
37mWhat I learned from monitoring more than 30 Machine Learning Use Cases
05mLightning Talk: Machine Learning Observability with Prometheus
18mReliable Maintenance of Machine Learning Models
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise
39mAn SRE Approach to Monitoring ML in Production
42mMachine Learning In Observability Systems: From Insights To Action
40mNext-generation Alerting and Fault Detection
26mMonitoring models in production