The ML Monitoring Flow for Models Deployed to Production
summary
Models degrade in production, median ~20% error increase at 365 days and some >80%, so reliance on tests fails; univariate drift tests produce many false positives while multivariate methods miss joint shifts. The workflow uses CBPE (confidence based performance estimation) and Direct Loss Estimation (DLE) with calibration (isotonic or LightGBM) and multivariate drift detectors (PCA reconstruction) and an open source NML implementation plus a cloud pipeline with alerts. On ~600 datasets CBPE outperforms the naive test set baseline, and TAPE addresses large covariate shifts.
AI-written summary · about
chapters (16)
- 0:00ML monitoring introduction
- 1:22Post-deployment performance challenges
- 3:09Model degradation in production
- 6:00Data drift overview and limits
- 9:52Univariate drift detection flaws
- 11:40Drift detection on real datasets
- 15:29Performance without ground truth
- 18:19Intuition for performance estimation
- 22:52Probability calibration role
- 26:44TAPE algorithm for performance
- 27:47Regression and other metrics
- 29:39Q&A: Retraining and drift segmentation
- 31:01Q&A: Calibration methods
- 32:27Q&A: Supervised vs unsupervised
- 33:11Q&A: Monitoring model drift
- 34:18Q&A: Seasonal inputs handling
related talks
39mAn SRE Approach to Monitoring ML in Production
19mModel Performance Monitoring - A Practitioner's Perspective
16mMLOps — Understanding and Fighting Model Degradation in Production
30mDetect machine learning (ML) model drift in production
31mMonitoring machine learning models in production
18mReliable Maintenance of Machine Learning Models
42mTesting Machine Learning Modules in Production
47mMachine Learning in Production: An Intro to MLOps.