Beyond A/B: Test automation approaches for web variants from A to Z
summary
trivago runs roughly 100 concurrent AB variants and shows that exhaustive end‑to‑end coverage is impossible; engineering builds a release controller, schedules weekend runs that iterate variants, and detects major revenue regressions within 24 hours. Shift‑left fixes use Storybook plus Playwright visual comparisons and developer unit tests; backend microservices get Karate GraphQL tests. Result: a hybrid strategy that drops the brittle decorator framework, automates variant activation for specific scenarios, and keeps regressions manageable.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:09AB testing automation
- 3:36AB testing importance
- 5:00Automation challenges
- 7:51Flaws in automation
- 10:22Holistic test automation
- 11:45Multiple variants testing
- 13:53Shift left testing
- 16:01Unit and visual tests
- 17:47Shift right monitoring
- 19:32Data-driven releases
- 21:17Variant regression handling
- 22:42Weekend testing tasks
- 23:25Q&A
related talks
43mLean QA: Automation, Monitoring, Experiments
28mOptimizing your A/B testing roadmap
55mShifting the Testing Role Pendulum
58mTesting Strategy - New Model, Better Outcomes
40mEnd to End Automation from a Developer Standpoint
47mFull-Stack Test Automation
53mHow we reconstructed our releases at Trivago
51mAutomating to augment testing