Experiments, the Good, the Bad, and the Beautiful
summary
Argues for replacing grand "experiments" with tiny actionable trials, showing why randomized science is usually impossible in orgs and why small simple fast and frugal trials work better; warns about placebo effects (roughly 50% improvement in placebo groups), sunk-cost bias, the backfire effect, and bad A/B practices that lack hypotheses; gives concrete moves like a time-boxed two-week standup trial, probe‑sense‑respond sequencing, Menlo/“open Friday” as accidental wins, and using humility over one-off proof.
AI-written summary · about
chapters (29)
- 0:00Intro
- 2:12Good stories
- 3:37Experiments
- 6:42B Tests
- 8:12Experiment
- 10:11Message
- 10:56One trial doesnt prove anything
- 11:55The placebo effect
- 15:09What can you do with trials
- 16:25Small is important
- 18:05Why simple
- 18:58Transparency
- 21:53Timebox
- 23:20The Bad
- 24:24Better Ways
- 25:12Context Adapt
- 26:12Probe
- 27:50Alexander Fleming
- 29:42SIPC
- 30:41Prepare to learn
- 32:31The innovators hypothesis
- 33:51The Hippos
- 34:53The Biggest Danger
- 35:13Awareness
- 40:01Dont argue
- 40:47Culture of experimentation
- 41:47Long list of experiments
- 43:41Big Problems Dont Have Big Solutions
- 45:54How do you differentiate from the Toyota Kata philosophy
related talks
01h 00mThe Experimentation Mindset
24mEffective Product Experimentation: Using Data-Driven Techniques for Iteration
39m2016 Keynote Linda Rising
09mInterview with
28mThe lean architect: Experimenting your way to better decisions
47mDoing I.T. for SCIENCE! - Sprints, Startups and the Scientific Method
01h 44mThe Power of Retrospection
35mExperimentation for Engineers