AB Test Bloody Everything
summary
Shows how to implement AB tests with a third‑party SDK (example: “generic AB testing”), using metadata-driven variants to change paywall text, colors or product lists without app updates; demonstrates measuring lift (example scenario: control 10% → variant 13%), warns about external confounders, and how to stop or roll back live tests. Covers an indie flow (500 DAU) and a corporate flow (1M DAU) with up to four variants, split users into test groups, targeted cohorts, and unit/UI testing strategies.
AI-written summary · about
chapters (21)
- 0:00Welcome and introduction
- 0:43What are AB tests
- 2:51Why fine isn’t good enough
- 7:24Measuring impact with AB tests
- 9:08AB testing for indie devs
- 16:49Setting up the first test
- 18:56Using metadata for flexibility
- 21:47Handling errors and fallbacks
- 23:33Tracking test results
- 25:20Experiment-oriented programming
- 26:44Waiting for test results
- 27:46Handling test failures
- 29:31Rolling out the winner
- 31:21AB testing for large companies
- 33:29Splurge’s AB test setup
- 35:13Benefits of many users
- 38:44Targeted testing groups
- 40:31Legal and privacy considerations
- 42:35Testing the AB test system
- 44:42Recap and key takeaways
- 45:02Live audience vote
related talks
52mBuilding an A/B Testing Framework that works for Developers
36mA/B Testing in Production
36mA/B Testing in Production
28mOptimizing your A/B testing roadmap
34mA/B Testing You'll Like
15mA/B Testing: Test Your Own Hypotheses & Prepare to be Wrong
43mLean QA: Automation, Monitoring, Experiments
41mA/B Testing with Style