What Failure Taught Me About Building High-Stakes Models
summary
Models spend over a million dollars a day and initial training optimizes RMSE of revenue per click, which surprisingly rewards a zero‑profit predictor. The team builds a shared Diagnostics notebook (stability, bias, error distributions, red/amber/green thresholds), factorizes targets into conversion rate and revenue per order, restores clear human‑model handoffs instead of daily retrains, and finds those changes stop catastrophic losses and make monitoring and interventions actionable.
AI-written summary · about
chapters (16)
- 0:00First day at Hotels.com
- 3:03High-stakes model requirements
- 5:53Fairy tale data science
- 8:46Single error metrics fail
- 10:28Diagnostics for trade-offs
- 13:43Ignoring side objectives
- 16:49Rebuilding with diagnostics
- 19:40Models need domain intuition
- 22:48Revenue per click components
- 25:57Humans out of the loop backfire
- 28:43Fairy tale data science fails
- 29:32Q&A on competition
- 30:37Discussing failure with stakeholders
- 31:40Modern risk management
- 32:45Q&A on resources
- 33:51Journey and failures
related talks
25mWhy Good Models Fail After Deployment
18mBuilding Models Using Ad Sales as an Example
25mMaking smart investments: A framework for maximizing your ROI in technical decisions
42mMasterclass: The right path to ML
30mWhen Good Models Go Bad: The damage caused by wayward models and how to prevent it
22mImproving Reliability by Data-Driven Engineering & Culture Changes
48mPrinciples and Practices of Building Machine Learning Systems
27mModel Evaluation for Humans