A Dull Beginning for a Drama-Free Ending
summary
Projects fail from early oversights and a few concrete mistakes: a bank model boasted offline .9 vs online .35 because payments register three days late; a rental recommender ignores business rules about top-row cars; a planning model treats a 10-person error the same for a branch serving three people; a startup’s A/B winner is later revealed as overwritten. The fix is a short design doc (two-week draft) that forces goals, metrics, reporting paths and error analysis and even catches when a simple median baseline outperforms complex models.
AI-written summary · about
chapters (21)
- 0:00Introduction and conference love
- 1:30Poll on past project failures
- 3:14Documentation’s love-hate relationship
- 5:00Design docs as a game changer
- 7:08Bank’s failed ML reminder system
- 9:38Faulty measurement in production
- 11:24Rental company’s ranking miscommunication
- 12:49Staff planning metric mismatch
- 14:55Wasted effort on complex models
- 15:59Startup’s fake A/B test results
- 18:07Why projects fail without design docs
- 19:34How to start writing design docs
- 22:23Design doc structure and evolution
- 23:05Expert advice on design docs
- 24:51Career opportunities shoutout
- 25:33Q&A session begins
- 26:14Q&A: Design docs solving problems
- 28:26Q&A: Document adoption challenges
- 31:41Q&A: When is documentation finalized?
- 33:46Q&A: Resolving document conflicts
- 35:36Q&A: Convincing leadership to adopt docs
related talks
23mHow to fail AI implementations
40mDesigning Data Driven Projects: Beyond ML Experimentation
35mWhat Failure Taught Me About Building High-Stakes Models
30mWhen Good Models Go Bad: The damage caused by wayward models and how to prevent it
48mPrinciples and Practices of Building Machine Learning Systems
42mMasterclass: The right path to ML
25mWhy Good Models Fail After Deployment
27m10 Things That Can Go Wrong with ML Projects