A/B User Testing When Your CDN Cache Messes With Your Results
summary
Problem: CDN caching caused double cache misses when client-side A/B routing served different processed asset versions, inflating startup latency and hurting engagement. Solution: implement asset bucketing at ingest so each variant propagates independently through the CDN, build an experiment pipeline that processes both buckets on ingest, and use bootstrapping plus a Z-test to measure significance across millions of videos in production; the implementation is deployed and used at scale.
AI-written summary · about
chapters (9)
- 0:00A/B testing with CDN
- 1:27CDN breaks user tests
- 3:52Cache latency in video
- 6:20CDN routing affects tests
- 9:50Isolating CDN impact
- 13:00Content performance variations
- 15:32Statistical pitfalls
- 17:40Real-world A/B lessons
- 18:22Summary
related talks
33mA/B testing: performative experimentation at scale
34mThe underappreciated power of content invalidation
29mA/B and see: Improving User Experience with Experimentation
40mReal-time A/B: how to get metrics for experiments
10mEnhancing CDN Performance and Cutting Egress Costs in Large Video Libraries
29mDelivering Fast Experiences with AEMaaCS
16mImproving Video Delivery and Streaming Performance with...
14mGustav Ernberg von Heijne at performance.sync()