AUC is Worthless: Lessons to Transition From Academic to Business Data Science
summary
Asserts AUC is worthless for business decisioning and builds a concrete loan example: fill a confusion-matrix value table, derive expected utility per applicant, and plot constant-utility lines that are straight so the slope decides whether default is “accept everyone” or “reject everyone.” Shows prevalence shifting from 0.5 to 0.6 flips the default, optimal threshold sits where an ROC curve is tangent, and a sunk customer acquisition cost cancels, so optimize business utility not AUC.
AI-written summary · about
chapters (13)
- 0:00Introduction and background
- 1:13Mismatch in skills
- 3:50Data science pillars
- 7:11Loss vs scoring functions
- 9:58Bridging technical and business
- 11:24Understanding ROC and AUC
- 13:05Loan application example
- 17:23Calculating expected utility
- 22:31Optimal decision thresholds
- 25:01Macroeconomic effects on models
- 27:20Summary of business utility
- 29:41Recommendations for data scientists
- 30:48Q&A session
related talks
29mEvaluation Design
37mBehind the AI Curtain - Designing for Machine Learning Products
23mDelivering business success with AI
25mTranslating AI Capabilities Into Better Business Results
37mSelection bias: The elephant in the room
32mAlgorithmic UDAP
30mBuilding fairer models for finance
38mHow to Enhance Customer Targeting in Marketing