Speeding Event Processing to 1.6M/sec: Badoo’s Experience
summary
Description of migrating a near-realtime event-processing pipeline at Badoo from a Spark+Hadoop stack to ClickHouse to handle much larger volumes. Covers the problems of scaling and fault tolerance under growing load, the technology swap to ClickHouse, and the resulting hardware cost reduction (3×) and throughput increase (from 300k events/sec to up to 1.6M/sec).
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:28Why collect stats
- 5:22Data collection phase
- 9:39Data processing phase
- 16:05Switching to ClickHouse
- 22:46Event storage and aggregation
- 28:04System technical specs
- 31:58Data visualization and monitoring
- 32:59Summary
- 33:20Q&A
related talks
52mUsing Hadoop at Badoo
52mUsing Hadoop at Badoo
38mHow to handle 100k events per second
57mLessons in Building a System that Processes More than 70 Billion Events Daily
08mReal-Time Data Systems: Speed, Scale, and Resilience
19mLog Events @ Twitter: Challenges of Handling Billions of Events per Minute
48mHyper Speed! When Big Data Blooms
22mIt’s all about the db - a journey to accommodate 100k events per second