10K Metrics, 500 A/B Experiments and 500M p‑values Every Day — How Is That Possible?
summary
An in-house A/B experimentation platform built at Avito is described, explaining how it scaled from a 2018 prototype to a production system running about 4,000 experiments per year. The talk covers operating at scale: collecting roughly 10,000 metrics, managing 500 concurrent A/B tests, and computing 500 million p-values daily, focusing on the platform architecture and automation that make that throughput possible.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 5:40Decentralized data problem
- 14:07Storage migration
- 18:23Long A/B tests and variance
- 25:29Compute engine optimization
- 34:27Summary
- 35:32Q&A
related talks
40mReal-time A/B: how to get metrics for experiments
31mBuilding a Data Experimentation Platform Without a Platform Team
42mImproving Search @scale with efficient query experimentation
55mHow to Improve Your Game's Retention & Monetization Through Data-Driven Experimentation
45mA/B Testing From The Ground Up
35mBuilding Data-Intensive Teams
13mData for product
34mA\B Test Best Practices and Fun Case Studies from Deliveroo