Building an AB testing analytics system with R and Shiny
summary
Builds an end-to-end A/B analytics stack using R packages and Shiny: an open-source funneljoin package that computes first/any/first-after funnels and runs on remote tables via dbplyr, automated health checks that catch duplicate cookies, missing pageviews and multiple starts, and a Shiny dashboard plus an impact calculator that returns power estimates (example: detect ~4% lift in 18 days). Describes lessons from DataCamp (200+ courses) and from prior Etsy work (60+ experiments, ~500/year platform scale).
AI-written summary · about
chapters (20)
- 0:00Intro
- 0:52About Me
- 2:36What is A/B Testing?
- 3:36Life B.D. (Before DataCamp)
- 4:17First weeks at DataCamp
- 5:23Questions I might answer about an A/B test
- 5:35Lengthy, repetitive code
- 5:57And when you're doing repetitive tasks...
- 7:08Structure
- 7:40Example: first-any
- 8:41Example: man-cap argument
- 9:18Things that have happened...
- 9:40You need to check your assumptions
- 9:56Initial solution
- 11:14By metric view
- 11:47Individual experiments view
- 13:03Impact cakulator
- 14:14Best Practice 1: Have one key metric per experiment
- 15:14Best practice 2: Run your experiment for length you're planned on
- 16:05Recap
related talks
20mBuilding a Flexible, Scaleable Self-Serve Reporting System with Shiny
15mA Robust Framework for Automated Shiny App Testing
33mAnalytics Data Driven Testing
46mAB Test Bloody Everything
52mBuilding an A/B Testing Framework that works for Developers
25mMake a Difference with Simple A/B Testing
11mFront-End Analytics for Smarter Products and Engineering Growth
18mData Science Careers via A/B Testing