A/B testing at Zalando: concepts and tools
summary
Overview of what actually happened and whether it’s useful tonight: Zalando describes real A/B infra with three services for bookkeeping assignment and analysis including deterministic hashing assignment, an open Python analyzer ExpAn, and a live fleet of 50 tests live aiming to scale to hundreds. Analysis runs automated t tests about t-test every six hours with Bayesian code present but too slow for production. Talks through concrete failures — multiple KPIs false positives, early stopping (peeking), and experiment conflicts — and ties results to expected business impact.
AI-written summary · about
chapters (15)
- 0:00Welcome
- 3:37Spurious correlations
- 6:31A/B tests
- 9:52A/B testing infrastructure at Zalando
- 11:58A/B Testing infrastructure at Zalando: software engineering perspective
- 16:10ExpAn: open-source library for analyzing A/B test results
- 16:58Hypothesis testing as a noisy binary classifier
- 18:00Example: is coin fair?
- 26:02Multiple KPIs / Endpoints testing
- 26:43Early stopping (peeking into tests)
- 28:02Going from data to statistics, from statistics to decisions
- 31:58Q&A
- 32:18Q&A: Do you use Classical tests or Bayesian tests?
- 33:45Q&A: What kind of experiments do you run?
- 35:08Q&A: What is the most egregious mistakes you have seen?
related talks
01h 15mAB-testing by cluster
41mA/B Testing with Style
43mA/B Testing: What to Pay Attention To
24mA B Platform Interactive Analysis in Realtime
44mA B Testing: Harder than just a color change
24mEffective Product Experimentation: Using Data-Driven Techniques for Iteration
38mEnough statistics so that Zed won't yell at you
42mA/B testing with real-world data to evaluate ML model updates