Of Mice and Metrics
summary
Compares lab and product work, noting only 10% vs 90% of data meets strict experimental standards and arguing useful evidence exists across a hierarchy. Claims a concrete hypothesis — better lists reduce churn — and describes concrete tools: event logging of filters and sorts, scoring qualitative feedback into numeric measures, randomized sampling instead of full A B rollouts, and statistical techniques like analysis of covariance to isolate effects.
AI-written summary · about
chapters (5)
- 0:00From research to software
- 1:37The scientific method
- 2:55Data hierarchy framework
- 4:23Practical application
- 6:55Refining experimental design
related talks
23mYou Get What You Measure—Why Metrics Are Important
27mMetrics that Matter - Moving from Easy to Impactful
01h 01mFinding Metrics that matter and using them safely
25mWhen Metrics Lie
37mYou Can't Optimize What You Can't Measure
01h 01mKeynote - Starting a metrics program
53mUsing Metrics to Take a Hard look at your code
47mWithout Data, you're just another Person with Opinions