TDI: a highly sensitive metric for A/B search experiments
summary
Presents TDI, a high-sensitivity metric for A/B experiments in search and ranking systems. Describes the problem that standard A/B metrics lack statistical resolution to detect improvements in ranking algorithms for sites like shops, job boards, and news aggregators, and proposes TDI as an approach to increase detection power for changes in search relevance.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 6:08A/B testing problems in search
- 14:46Team Draft Interleaving algorithm
- 22:19Implementation technical difficulties
- 31:57Bootstrap method for analysis
- 36:54Method application and development
- 40:59Q&A
related talks
24mOffline Ranking Validation - Predicting A/B Test Results
25mAI Reliability in Security: A New Data Science Paradigm
28mThe Hitchhiker's Guide to Advanced A/B Testing Techniques
38mA Bi-metric Framework for Fast Similarity Search
18mHow Data Scientists Broke A/B Testing (And How We Can Fix It)
41mDynamic Ranking of Search and Recommendation Results in High-Traffic Systems
03mDBTB INT Scott Clark r
18mData-driven Decision Making in Ad-Tech