Nothing to Recommend It: An Interactive ML Outage Fable
summary
An online yarn shop uses ML to rank search and surface recommendations, retraining the same model daily retraining and deploying unchanged config. Customers see weird search results and finance reports a 5–10% revenue drop. SREs sample 100,000 user queries (creating privacy risk) and confirm serving and model code are unchanged; root cause is demand shift to lightweight yarns plus supplier stockouts and an out-of-stock filter that hides unavailable items, so the model is working but the catalog cannot satisfy demand.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 3:24Agenda
- 4:03Who am I
- 5:27Yarnit
- 7:36Story
- 10:36Audience Participation
- 11:27Questions
- 14:29The Answer
- 15:07What Do We Do
- 18:45What Does It Mean
- 20:01What Will They Find
- 21:16Privacy Interlude
- 21:35Privacy Infrastructure
- 24:36What actually happened
- 26:13The correct answers
- 26:36Takeaways
- 28:20Reliable Machine Learning
- 28:52Outro
related talks
40mDesigning Data Driven Projects: Beyond ML Experimentation
48mPrinciples and Practices of Building Machine Learning Systems
24mChallenges in building Machine Learning models in production
19mHow ML Breaks: A Decade of Outages for One Large ML Pipeline
23mHow Can We Trust Machine Learning? Exploration, Evaluation...
24mWhen Recommendations Systems Go Bad
59mWhy Great Models Fail: Lessons From 9 Years of Deploying ML Models
41mBuilding MLOps Infrastructure at Japan's Largest C2C E-Commerce Site