Experimentation and the gold standard of data champions
summary
Claims that companies can run 10,000 experiments simultaneously and get reproducible causal answers, with tooling built around feature flagging and a progressive disclosure UI that shows lifts, confidence intervals and a day‑since‑first‑exposure chart. Recommends standardized metrics, automation, health checks, and overlap experiments instead of carving users up. Cites big‑tech hit rates (roughly 10–33% winners) and a paper claiming adopters see 30–100% lift over a year. Offers concrete engineering advice: plumbing metrics, defaults, and making experimentation an engineering practice.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 2:28What experimentation is
- 4:15Product analytics hierarchy
- 6:20A/B testing gold standard
- 8:07Cultural transformation
- 11:24Limitations of dashboards
- 14:11Scaling experimentation challenges
- 15:34Product observability
- 16:58Modern experimentation culture
- 19:23Walk Jog Run Fly framework
- 20:48Experimentation best practices
- 24:18Experimentation in startups
- 27:48Overlapping experiments
- 30:16Data champions best practices
- 32:01Scaling experimentation
- 33:05Q&A
- 35:32Q&A: Multiple metrics
- 37:36Q&A: Data collaboration
related talks
11mFrom 0 to 10,000 experiments
24mEffective Product Experimentation: Using Data-Driven Techniques for Iteration
40mPanel Discussion: What Users Really Want
52mAB-tests, lies, damned lies, and statistics.
20mHow to Win Friends and Influence People (With Data)
50mData-Driven Engineering
48mStanding Up for the Unmeasurable in The Design Process
42mThe 7 habits of data effective companies