A/B and see: Improving User Experience with Experimentation
summary
A zip‑code placement experiment shows a tiny reorder becoming a large win: variant 3 moved zip code under last name increases signups while variant 4 removed the zip field crashes conversions because geolocation guesses are often wrong. Netflix runs server‑side allocations, a codecs service that builds custom bundles, AA and holdback tests (99% rollout with 1% holdback) and quasi‑experiments by region. Optimizely Adobe and Google Analytics are viable starter tools.
AI-written summary · about
chapters (27)
- 0:00Intro
- 4:40Agenda
- 5:07What is an AB test
- 6:41Data
- 7:32Goals
- 8:36Hypothesis
- 9:58Variations
- 10:27Sample Size
- 11:16Split Equally
- 12:22serialize
- 14:14start small
- 14:32run simultaneously
- 15:35analyze results
- 16:22what can go wrong
- 16:37let a be test run
- 17:27look at the correct variables
- 18:08retest often
- 18:56example
- 19:51choose the right metrics
- 20:19scheduling complexities
- 20:45code complexity
- 21:22experiment at Netflix
- 21:57JSON payload
- 23:07AAA test
- 23:32Holdback tests
- 23:59Quasi experiment
- 24:35Wrap up
related talks
19mA/B User Testing When Your CDN Cache Messes With Your Results
27mA Tale of Two Components: Mastering A:B Testing in React
09mDemystifying the Dreaded A/B Test
45mHow we killed 80% of features and increased outcomes of a/b testing by 100%
31mHow to think `User first`
28mBeyond A/B: Test automation approaches for web variants from A to Z
38mA/B testing and feedback techniques in your mobile apps
38m"Performance Impact, Part Two: More Findings from the Front Lines..."