How we killed 80% of features and increased outcomes of a/b testing by 100%
summary
A/B testing is measured here with a year-long holdout group that freezes 10% of users to capture cumulative and long-term effects; that revealed a temporary -4.5% revenue hit from a stats-pipeline bug, then a new regime that cuts scaled inconclusive tests from 70% to 40% and doubles net A/B value. Technical tweaks matter: swapping to ujson change dropped revenue 11% and a cache fix caused ~3% GMV loss, so an in-house system ties Sentry/Elasticsearch/Datadog signals to auto-rollbacks.
AI-written summary · about
chapters (29)
- 0:00Introduction
- 2:05Question
- 3:29About the company
- 4:52History of Starship
- 6:08From the commercial point of view
- 6:59The full spectrum of emotions
- 9:34What is AB testing
- 11:50Questions
- 12:56How we can measure the whole effect of the positive test
- 14:55Pros Cons
- 16:06Product Development
- 16:26Guesses
- 17:53Results
- 18:50How we iterate
- 19:10Neutral experiments
- 20:38Incremental boost
- 21:14No neutral experiments
- 21:39Inconclusive experiments
- 22:15Scale ratio
- 23:33Pull requests
- 24:06UJSON
- 24:56Latency
- 25:22Share
- 28:25The system
- 28:49Data collection
- 29:32DDoS protection
- 31:21Launching more ab tests
- 32:00Extreme experiment distribution
- 37:48Team
related talks
28mOptimizing your A/B testing roadmap
09mDemystifying the Dreaded A/B Test
38mTesting, Testing: On Experimental Drift and Data Driven Product Design
32mAB Testing to Determine Product Priority
33mHow to A/B test with confidence
18mData Science Careers via A/B Testing
27mBehind the Curtains: A/B Tests in Video Land
36mHidden engineering challenges of A/B Testing