The Scientific Method of Experimentation
summary
Explains how to run A/B experiments like science: form a testable hypothesis, do prior usability work, define success and guardrails, run a controlled launch and analyze p-values, confidence intervals and practical significance; includes concrete design choices (targeting, triggering, session units) and lessons on novelty, learnability and false positives. Gives real examples from Google and Amazon and prescribes processes and templates, including a concrete Prime filter toggle on iOS, a 2-week weekly cadence, and a revenue + latency guardrail.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 0:33What is experimentation
- 1:22What is user research
- 2:57What does experimentation give
- 3:38What is an experiment
- 5:45Why launch an experiment
- 8:31The scientific method of experimentation
- 9:20Product management
- 11:09Amazon example
- 12:49Design the experiment
- 14:10Triggering criteria
- 17:56Practical significance
- 19:44Draw conclusions
related talks
01h 03mExperimentation in Data Science
34mA/B Testing You'll Like
27mHow to A/B test with confidence
15mA/B Testing: Test Your Own Hypotheses & Prepare to be Wrong
18mHow Data Scientists Broke A/B Testing (And How We Can Fix It)
28mThe Hitchhiker's Guide to Advanced A/B Testing Techniques
09mA/B tests: Understanding user behaviour
29mBalancing Personalization & A/B Testing in a Customized World