Testing Machine Learning Modules in Production
summary
FINRA builds and deploys a monitoring stack that: measures model quality with ROC AUC, detects data shifts with total variation and earth mover’s distance, and runs a lightweight divergence tool that flags days with unusually high drift (July 25th shows ~0.44 total variation and reveals a missing upstream feed). The system also uses an exploration/exploitation thresholding policy (reinforcement-learning style) to request human labels, cutting labeling waste from ~45% toward ~0.5% over 3,600 rounds while catching issues like flipped buy/sell codes.
AI-written summary · about
chapters (12)
- 0:00Testing ML in production
- 1:31Traditional testing fails for ML
- 4:44Data engineering in ML
- 6:07Validating training data
- 7:32Feature importance sanity check
- 9:59Quantifying ROC AUC
- 12:01Monitoring performance over time
- 14:31Detecting data shifts
- 19:44Divergence detection example
- 22:53Setting alert thresholds
- 26:03Balancing thresholds
- 31:40Question and answer session
related talks
23mChallenges in using machine learning for financial monitoring
13mHuman-in-the-Loop MLOps: Boost Model Performance by 40%
36mThe ML Monitoring Flow for Models Deployed to Production
39mAn SRE Approach to Monitoring ML in Production
31mMonitoring machine learning models in production
36mContinuous Monitoring of Machine Learning Models
18mReliable Maintenance of Machine Learning Models
19mModel Performance Monitoring - A Practitioner's Perspective