How Etsy Handles “Peeking” in A B Testing
summary
Etsy implements a Python-based sequential testing system that replaces fixed-horizon A/B tests with a p-value curve approach derived from Evan Miller’s sequential simple AB tests, computing adjusted stopping thresholds on converted visits to keep overall false positives near 5%. Simulations using five years of historical experiments show early stops typically save at least a week for multi-week tests, while small effects (<5%) tend to be inflated; UI changes surface power, CI visuals, and messages like “directional change but magnitude might be inflated.”
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:26About Etsy
- 0:55Etsy Stats
- 1:24What is AP Testing
- 2:48Peeking
- 4:40Example
- 5:29Solution
- 6:33Platform
- 7:22Sequential Testing
- 12:05Power Confidence Interval
- 18:35Conclusion
related talks
38mTesting, Testing: On Experimental Drift and Data Driven Product Design
45mHow we killed 80% of features and increased outcomes of a/b testing by 100%
47mHow to be a Performance Detective?
25mMake a Difference with Simple A/B Testing
33mHow to A/B test with confidence
32mAB Testing to Determine Product Priority
28mOptimizing your A/B testing roadmap
09mDemystifying the Dreaded A/B Test