A/B Testing with Style
summary
Walks through a working A/B system called Jacquard and why it replaces per-user DB rows with 300 buckets hashed from user IDs, so tests need no signup writes and page hits avoid DB reads. Shows a 1% canary rollout then a 50/50 experiment, and uses Bayesian posteriors (priors ~20–30%) to report probability distributions of lift rather than a binary decision rule. Covers practical risks: confirmation bias, survivorship, inertia and ethics.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:06What is A/B Testing?
- 2:40Why would you do it?
- 5:13How do you do it?
- 8:13Old way
- 11:10New way
- 13:46Demo - jacquard
- 20:45How to measure test data?
- 22:50No decision rule
- 25:57Risks
- 29:50Ethics
related talks
36mA/B Testing in Production
36mA/B Testing in Production
38mUsable A/B testing – A Bayesian approach
41mLittle-Known Pitfalls of A/B Testing
19mBeing Intentional: AB Testing and Data Minimization
27mCustomer Driven Development : What, Why & How?
37mA/B testing at Zalando: concepts and tools
25mScaling Bayesian Experimentation