Continuous Monitoring of Machine Learning Models
summary
Zalando Payments monitors deployed fraud models because labels arrive with a two-week label delay, so accuracy alerts are too late. The system captures live features via an SQS lumberjack pattern, summarizes feature CDFs with TDigest summaries, compares live vs test distributions (KS-like distance) on a 12-hour sliding window, and triggers alerts when distances exceed thresholds. Models serve predictions within one second, Spark batches aggregate digests, AWS Data Pipelines automate jobs, and the approach successfully detects unit/format shifts before revenue impact.
AI-written summary · about
chapters (5)
- 0:00Introduction to Zalando
- 6:05Importance of ML monitoring
- 14:36Monitoring strategies
- 21:34Implementation details
- 30:38Best practices and summary
related talks
37mThe Role of Machine Learning in Today’s Payment Ecosystems
19mModel Performance Monitoring - A Practitioner's Perspective
39mFraud Detection: Using ML to Identify and Manage Fraudulent Activities
42mTesting Machine Learning Modules in Production
18mReliable Maintenance of Machine Learning Models
23mChallenges in using machine learning for financial monitoring
26mMonitoring models in production
25mSymptom-based Alerting for Machine Learning - What I Learned...