Challenges in building Machine Learning models in production
summary
Practical account of deploying personalization on booking.com: concrete failure modes, fixes and measured effects. Covers continuous cold start and using content models (topic detection over reviews, content evaluation icons) to help anonymous users, a classifier that reaches 70% accuracy but is actually memorizing city-level supply, and the position bias problem solved via costly randomized rankings. Emphasizes running ~1000 online experiments to validate interventions and iterating until an experiment shows a real user-facing lift.
AI-written summary · about
chapters (8)
- 0:00ML model challenges in production
- 1:47User behavior and cold start
- 6:18Data limits and label ambiguity
- 10:11Dynamic data and feature compression
- 12:40Position bias in ranking models
- 15:12Intent vs observed outcomes
- 17:37Live experimentation necessity
- 19:00Q&A
related talks
08mConsiderations for producing the best AI and Machine Learning models
22mBefore the Model: How Machine Learning Products Start...
33mRunning Machine Learning in Production - a Journey to Success
48mPrinciples and Practices of Building Machine Learning Systems
40mMachine Learning
59mWhy Great Models Fail: Lessons From 9 Years of Deploying ML Models
39mChanging Realities - The Final Frontier for Machine Learning?
36mApplied Machine Learning Playbook: From Idea to Production