Monitoring models in production
summary
Monitors Keras neural nets in production, shows a concrete pipeline: serve predictions, send inputs and outputs to a separate watchman instance that stores transactions, computes entropy and distributional drift (KL/absolute deviation) and triggers alerts. Uses Flask and Docker for serving, mentions Prodigy/Spacy for human labeling, and cites active learning that can cut labeling needs by 40X. Includes code snippets for model serving, a practical alert example (50% more “parks” flagged), and lightweight open‑source tooling.
AI-written summary · about
chapters (15)
- 0:00Economics to ML in production
- 1:50Why model monitoring is essential
- 3:56Model decay in production
- 5:23Active vs online learning
- 7:10Challenges in monitoring behavior
- 8:35Monitoring inputs for data drift
- 9:39Monitoring outputs and confidence
- 10:41Ground truth labeling
- 12:51Alerting and dashboards
- 15:21Real-world impact
- 17:33Societal impact
- 19:39User feedback loops
- 20:00Open-source example
- 22:33Future of data science
- 24:07Q&A
related talks
31mMonitoring machine learning models in production
25mMonitoring AI with AI
18mReliable Maintenance of Machine Learning Models
19mModel Performance Monitoring - A Practitioner's Perspective
39mAn SRE Approach to Monitoring ML in Production
37mWhat I learned from monitoring more than 30 Machine Learning Use Cases
25mSymptom-based Alerting for Machine Learning - What I Learned...
35mGetting Machine Learning Models Ready For Production Using Python