Popmon: Population Shift Monitoring Made Easy
summary
A lightweight Python pipeline called popmon monitors covariate shift by histogramming features, applying tests like chi-square and Pearson/KS, and producing offline HTML reports and stored metrics for pandas or spark workflows. ING uses statistical process control bounds, dynamic traffic lights, and stitching of batches; a detected peak in September 2018 exposes a data‑source switch and retraining only on the new source materially improves model performance, showing the system finds actionable distribution shifts before late validation.
AI-written summary · about
chapters (14)
- 0:00Population shift monitoring
- 1:55Why monitor shifts
- 4:20Data issues and splits
- 6:07Limitations of profiling
- 8:14How Popmon processes data
- 10:00Statistical tests for shifts
- 11:48Histogram-based monitoring
- 13:14Popmon use cases
- 14:40Reference points and alerts
- 17:52Generating Popmon reports
- 21:01Live demo
- 27:18Custom monitoring pipelines
- 29:26Internal use case impact
- 30:50Future plans and contributions
related talks
28mOnline Change Point Detection Using Spark Streaming
12mModel Monitoring: Detecting and Analyzing Data Issues
18mReal-Time Monitoring of Distributed Systems
26mMonitoring models in production
01h 21mBuilding a ML Model Observability pipeline
18mPV Monitoring Based on Linear Regression
19mModel Performance Monitoring - A Practitioner's Perspective
17mStopping Performance Regression via Changepoint Detection