The Realities of A/B Testing
summary
A practical playbook for running many A/B tests: use Firebase Remote Config for feature flags, Google Analytics / BigQuery plus Python for results, and a secret debug panel plus UI tests to flip flags. Build fast prototypes as stretched screenshots (worked in production), accept short-lived duplicated code rather than complex abstractions, and clean up immediately. Watch out for crashes when flags change after UI loads. Example results: 20+ concurrent flags, 10% audience splits, and a personalization test that lifts purchases ~30%.
AI-written summary · about
chapters (10)
- 0:00Introduction to A/B testing
- 1:50What is A/B testing
- 3:57Cost and strategy of A/B tests
- 8:39Implementing A/B tests
- 13:23Learnings from feature flags
- 22:06Prototyping shortcuts
- 27:16Testing and cleanup
- 28:01Analyzing test results
- 32:50Real-world A/B test examples
- 34:44Summary and key takeaways
related talks
30mStats Don't Have to Be Scary: Automatic a/b Test Analysis Using Python
32mTurn Your App Data into Answers with Firebase and BigQuery
33mAnalytics Data Driven Testing
44mA Crash Course in Google Analytics for Firebase and BigQuery
22mSupercharge your app with Firebase and BigQuery
34mMaximizing marketplace experimentation: switchback design for small samples
46mFirebase & BigQuery - Do Mobile App Analytics Easily & at Scale - Queries Included
29mCrash-reporting taken to next-level