Log Events @ Twitter: Challenges of Handling Billions of Events per Minute
summary
Pipeline scales to 5 billion events per minute and tens of petabytes per day by replacing Scribe with Apache Flume plus memory, micro‑batching and service‑disintegration optimizations to eliminate CPU/memory/disk bottlenecks so network is the limiter. Hourly HDFS + MapReduce processing moves to DAGs with Apache Beam + Google Dataflow/Tez and dynamic hash partitioning, plus data tiers and aggregator groups for per‑dataset scaling. A streaming path writes to Google Pub/Sub and BigQuery streaming ingest, cutting end‑to‑end latency from hours to seconds or sub‑second for small workloads.
AI-written summary · about
chapters (17)
- 0:00Event log pipelines at Twitter
- 1:08Scale of the event log pipeline
- 2:12Original architecture limits
- 3:55Aggregation layer scaling
- 5:44Optimizing Flume throughput
- 6:49Data tiers and grouping
- 7:54Dynamic aggregator scaling
- 9:40Processing bottlenecks and fixes
- 11:27Migrating to Tez
- 12:51Five billion events per minute
- 13:33Reducing end-to-end latency
- 14:36Streaming aggregation design
- 15:36Stream processing with Beam
- 16:41Real-time analytics at scale
- 17:44Latency results and use cases
- 18:05Future streaming challenges
- 18:50Acknowledgments
related talks
24mGive meaning to 100 billion analytics events a day, analytics at Teads
55mCloud Analytics: from 0 to 100,000 events per second in 50'
41mGoogle Dataflow: New open model for batch & stream processing
50mHow We Store 60,000 Events per Second
41mFrom Batch to Realtime with Hadoop
20mStreaming Processing Options in Google Cloud
11mBig Data Analytics: Immutability & Idempotence
43mA Data Streaming Architecture with Apache Flink #bbuzz