Experimentation at scale: Fuel your application’s growth through A/B testing
summary
Argues that fast build–measure–learn cycles come from disciplined A/B testing and feature flags, shows a runnable Android example using shared preferences and Firebase Remote Config, warns that many variants blow up statistical power (Google’s 41 blues gives ~88% false‑positive risk), and explains why Lyft replaced third‑party SDKs with an internal TCS target configuration service that ties experiments to bespoke metrics, supports fine geographic targeting, exposes dev panels on device, runs >1,000 concurrent experiments across ~120 apps and scales with 2,000 modules and weekly releases.
AI-written summary · about
chapters (13)
- 0:00Introduction to growth
- 1:29Why test with A/B tests
- 5:05How A/B tests work
- 13:50Commercial A/B testing frameworks
- 17:22Firebase A/B testing basics
- 27:26Avoiding A/B test pitfalls
- 33:51Statistical significance explained
- 39:12A/B testing at Lyft
- 46:20Clean architecture for A/B tests
- 58:20Lyft’s custom A/B testing tool
- 1:03:12Debugging with feature flags
- 1:04:14Timeline of Lyft’s A/B testing
- 1:07:02Q&A on A/B testing decisions
related talks
27mExperimentation in BigTech Validating Hypotheses and Driving Innovation
52mBuilding an A/B Testing Framework that works for Developers
34mA/B Testing You'll Like
46mTweaking and AB Testing Your App on the Fly
28mOptimizing your A/B testing roadmap
17mIterate your way to a delightful app experience using Firebase Remote Config and A/B Testing
46mTowards Joyful Development
24mEffective Product Experimentation: Using Data-Driven Techniques for Iteration