When Good Models Go Bad: The damage caused by wayward models and how to prevent it
summary
Hand analysis of ~100 incidents over 10 years shows many failures are infrastructure handovers not modeling math. A large ranking pipeline with thousands of models, 100 billion parameters retrained hourly produces subtle bugs: a safe model is accidentally assigned an experiment ID that excludes its logs, ramps from 1% to 100% and starves training data, triggering slow alerts. Proven mitigations: enforce ID ranges, monitor data volumes and feature distributions, auto-pause on quality drops, and keep searchable postmortems.
AI-written summary · about
chapters (9)
- 0:00Introduction and speakers
- 1:31Who we are
- 3:20Model quality as production problem
- 6:29System context and architecture
- 10:44Taxonomy of model failures
- 16:05Case study of a failure
- 20:38Prevention and protection strategies
- 22:48Learning from failures
- 25:34Q&A session
related talks
25mWhy Good Models Fail After Deployment
30mFrom Black Box to a Known Quantity: How to Build Predictable, Reliable ML-based
28mWhy we do ML model retraining wrong, and how to do better
18mReliable Maintenance of Machine Learning Models
35mWhat Failure Taught Me About Building High-Stakes Models
36mHow to review a model
19mHow ML Breaks: A Decade of Outages for One Large ML Pipeline
01h 04mCan you trust your (large language) model?