A/B Testing From The Ground Up

on YouTube ↗45m

" summary

Yelp’s Android experimentation platform uses repeatable bucketing by hashing an identifier with an experiment-specific salt to avoid correlated assignments, supports parameterized experiments via Firebase Remote Config or Facebook PlanOut to simplify complex UI combinations, and persists client events to a tape queue + job manager (flush every 30s or 20 events, immediate flush on background) for delivery; internal analysis exports to BigQuery, and measured delivery loss is about 0.3%, an accepted trade-off.

AI-written summary · about

" chapters (12)

1. 0:00Introduction to A/B testing
2. 1:22Why run A/B tests
3. 5:34Key A/B testing infrastructure
4. 9:04Bucketing system pitfalls
5. 11:12Bucketing system implementation
6. 13:17Parameter-based experiments
7. 17:08On-device logging principles
8. 24:07Event deliverability challenges
9. 29:59Analyzing experiment results
10. 33:08Internal vs third-party metrics
11. 37:14Takeaways and resources
12. 39:17Q&A session

" related talks

• 55mHow to Improve Your Game's Retention & Monetization Through Data-Driven ExperimentationNaman Jain · Indie Game Business Sessions: Winter 2020
• 28mOptimizing your A/B testing roadmapJean-Loup Yu, Vesna Doknic, Karin Tsai, Bruno Vaz Moço · Playtime 2019
• 24mMobile ExperimentationBen Bader · Mobile @Scale 2023
• 33mAndroid Benchmarking and other storiesEnrique López Mañas, Iury Souza · droidcon New York 2022
• 25mMake a Difference with Simple A/B TestingDanielle Gordon · RailsConf 2021
• 29mAndroid Benchmarking and other storiesEnrique López Mañas, Iury Souza · droidcon Berlin 2022
• 43mLean QA: Automation, Monitoring, ExperimentsKateryna Chernikova · Selenium Camp 2019
• 34mA\B Test Best Practices and Fun Case Studies from DeliverooJamie Adkins, Edward Harker · Droidcon London 2023