Using A B testing to rebuild the UI used by 1 5 million users a day
summary
Skyscanner rewrites the main flights search widget used by 1.5 million daily users and runs A/B testing from day one to validate it. Teams follow the Spotify squad model, swap the component, and discover three concrete failures: rolling out more than one independent variable so results are uninterpretable, an instrumentation bug that misses search-click timing and flips results until fixed, and no predeclared success metric so signal looks noisy; after fixing tracking conversion improves but other KPIs dip and rollout pauses.
AI-written summary · about
chapters (4)
- 0:00Start
- 0:46Spotify Squad Model
- 2:12Testing Pitfall We Put Way Too Much Trust into Our Instrumentation
- 4:39Define Your Success Criteria up Front
related talks
52mBuilding an A/B Testing Framework that works for Developers
28mOptimizing your A/B testing roadmap
34mA/B Testing You'll Like
28mExperimenting your way to a better product
52mAB-tests, lies, damned lies, and statistics.
28mBeyond A/B: Test automation approaches for web variants from A to Z
45mHow we killed 80% of features and increased outcomes of a/b testing by 100%
36mA/B Testing in Production