An SRE Approach to Monitoring ML in Production
summary
Claims that 91% of models degrade and that 50% of teams run no ML monitoring drive concrete practices: monitor four layers—infrastructure, service, data, model—track data freshness, schema, bias and distribution drift using tests like Kolmogorov–Smirnov and KL divergence, record model metadata (who/when/size), add CI/CD and canary routing for model versions, and alert on rising validation vs production error. Note that CI/CD for models is still immature and training loops must be automated to retrain when drift exceeds thresholds.
AI-written summary · about
chapters (13)
- 0:00SRE for ML monitoring intro
- 3:53Why ML models degrade
- 7:23SRE vs ML practices
- 10:11Cross-domain ML monitoring challenges
- 14:02SRE pyramid for ML workloads
- 18:16Monitoring ML data pipelines
- 21:03Training ML at scale
- 26:23ML production pipelines
- 29:09Monitoring ML systems
- 31:57Data drift detection
- 34:22ML observability antipatterns
- 36:08Key takeaways
- 37:33ML practitioner recommendations
related talks
16mMLOps — Understanding and Fighting Model Degradation in Production
18mReliable Maintenance of Machine Learning Models
44mSRE and ML: Why It Matters
36mThe ML Monitoring Flow for Models Deployed to Production
44mSRE for ML: The First 10 Years and the Next 10
47mMachine Learning in Production: An Intro to MLOps.
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
29mDemystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform