It’s all about the db - a journey to accommodate 100k events per second
summary
System ingests 100k+ events/sec end to end, tested through three DB stages: Redis: a few thousand/sec, limited by single-threaded I/O and round trips, TimescaleDB: works to ~30k/s with hypertables and continuous aggregates but inserts drop at ~100k/s and aggregations take hours, ClickHouse: bulk inserts, monthly partitions, SSD→S3 tiers, single server with 30s queries and handling >150k/s. Uses async Python with batched I/O, Tenacity for retries, and separates ingestion pods from CPU-heavy processing.
AI-written summary · about
chapters (6)
- 0:00Introduction and background
- 3:23Starting with Redis
- 7:18Moving to TimescaleDB
- 13:43Switching to ClickHouse
- 19:47Outcomes and lessons learned
- 21:12Q&A
related talks
26mHandling Large Volumes of Immutable Structured Data With ClickHouse
17mUsing ClickHouse® Database to Power an Analytics and Customer Engagement Platform
50mHow We Store 60,000 Events per Second
48mHyper Speed! When Big Data Blooms
42mClickHouse: what is behind the fastest columnar database
28mColumnstore Revolution
24mScaling to Billions of Internet Observations per Day
47mSpeeding Event Processing to 1.6M/sec: Badoo’s Experience