Deduplication of 5 Million Events per Second on YDB in AppMetrica

on YouTube ↗37m

" summary

Describes AppMetrica’s event-processing pipeline built to deduplicate 5 million events per second using YDB. Covers the problem of massive-event deduplication, the overall architecture of about 50 microservices communicating via ClickHouse and ZooKeeper, and how the pipeline handles those requirements.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 7:26Requirements
3. 12:27First cloud implementation
4. 13:49Performance issues
5. 14:32Bloom filter optimization
6. 18:06Current architecture and metrics
7. 20:36Service scaling
8. 22:25Event-time optimization
9. 23:51Q&A

" related talks

• 47mSpeeding Event Processing to 1.6M/sec: Badoo’s ExperienceAleksandr Krasheninnikov · HighLoad++ Moscow 2018
• 26mImplementing YDB CDC: The Yandex Monitoring CaseEgor Litvinenko · Saint HighLoad++ 2023
• 42mPetabytes in YDB over HDD in Yandex.Metrica ProcessingAnton Barabanov · HighLoad++ 2023
• 48mHyper Speed! When Big Data BloomsScott Helme · NDC London 2023
• 22mIt’s all about the db - a journey to accommodate 100k events per secondEyal Balla · PyCon Israel 2024
• 38mHow to handle 100k events per secondNiels Basjes · Berlin Buzzwords 2019
• 24mGive meaning to 100 billion analytics events a day, analytics at TeadsPar Alban Perillat-Merceroz · DataXDay 2019
• 01h 00mParallel Asynchronous Replication b/t YDB Database InstancesAndrey Fomichev, Ilnaz Nizametdinov · Hydra 2022