Experimentation in BigTech Validating Hypotheses and Driving Innovation
summary
Meta-style experimentation shows how a single APK with Firebase Remote Config and Meta’s internal Quick Experiments framework runs gated A/B tests, sampling events at about 1 in 10,000 and allocating user universes (example: 30% experiment split into 15% control / 15% test). A reproduced Android demo flips a blue-to-red button and a real experiment reports a sustained 1.5% CPU reduction over a week, with p‑values and 95% confidence intervals guiding rollout.
AI-written summary · about
chapters (14)
- 0:00Experimentation at Meta
- 1:50Experimentation vs A/B testing
- 2:56Experimentation with remote config
- 4:23Experimentation universe setup
- 6:32Success metrics and confidence
- 8:41Experimentation for innovation
- 10:06Experiments with Firebase
- 13:20Firebase project setup
- 14:43Remote config in Firebase
- 15:50Fetching remote config in code
- 17:19First experiment test
- 21:13Managing test devices
- 23:01Launching and analyzing results
- 24:03Q&A
related talks
01h 15mExperimentation at scale: Fuel your application’s growth through A/B testing
46mTweaking and AB Testing Your App on the Fly
39mImprove your app metrics in weeks with Play Store Experiments (en)
35mExperimentation for Engineers
24mEffective Product Experimentation: Using Data-Driven Techniques for Iteration
52mBuilding an A/B Testing Framework that works for Developers
28mOptimizing your A/B testing roadmap
13mPersonalization and A/B Testing: How to create better experiences for users