Preparing Yandex search data — the library and processes we built
summary
Yandex Search data-prep team describes building a processing pipeline that handles 5 Gb/s of streaming traffic. The talk explains why they chose a hybrid between lambda and kappa architectures, the library-based approach analysts use in SQL FROM clauses instead of tables, and details of the libraries and processes they implemented.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 2:34Search results evolution
- 5:40Log issues and data
- 9:11Query aggregation library
- 13:49Batch versus streaming
- 16:58Data uniformity
- 19:48Final architecture
related talks
39mHow Data Delivery Evolved for Yandex’s Ad Engine
46mA Look Behind the Scenes of Yandex.Maps: From Raw Satellite Data to Clear Images
41mReliability of Data Delivery in Stream Processing
41mAccelerating privacy-enhancing data processing
48mYTsaurus — The Future of DWH, Already Here in Yandex Market
50mHow We Built a Distributed Queue Service at Yandex
51mYandex Database: Distributed Queries in the Clouds
24mA Gentle Introduction to Building Data-Intensive Applications