Behind the Curtains: A/B Tests in Video Land
summary
Short-form video quality improvements increase engagement even when upload latency and failures initially worsen, forcing investment to fix reliability; a prototype shows users prefer higher quality and doubles engagement on those clips. Rolling out more efficient encodings causes a CDN cache miss problem — cache hit rate 20% vs 95% — fixed by adding a cache-breaking key to controls so tests are comparable, after which efficiency improves consumption. Practical rules: 5000 users for 30% lift vs 5M for 1% and ship changes behind feature gates/progressive rollout.
AI-written summary · about
chapters (10)
- 0:00Experimenting in video infra
- 3:15Why video for A/B tests
- 6:48How A/B tests changed video strategy
- 9:59Video encoding experiment challenges
- 12:28Modern experimentation without the overhead
- 14:11Debunking A/B test scale myth
- 16:59Q&A on experimentation at scale
- 22:16Small companies and A/B tests
- 24:27Choosing experiment metrics
- 25:30Future of experimentation
related talks
28mOptimizing your A/B testing roadmap
45mHow we killed 80% of features and increased outcomes of a/b testing by 100%
09mDemystifying the Dreaded A/B Test
33mHow to A/B test with confidence
35mBetter, Faster, Stronger - Delivering High Quality Products
48mAudio and Video Testing Fundamentals
36mHidden engineering challenges of A/B Testing
56mBetting on Performance: A note on Hypothesis-driven Performance Testing