Migration of a realtime stats product from Storm to Flink
summary
ResearchGate migrates a realtime stats pipeline from Storm to Flink, replacing per-event +1 DB writes with time-window pre-aggregation and keyed state, cutting ~1 billion daily increments to ~200 million (~80% fewer writes) and lowering backend load by more than 30%. Pipeline uses Kafka for ingestion and as a sink buffer, a dispatcher service to batch HBase writes, and optimistic locking for correctness. Replaces stateless increments with idempotent stateful counting, implements revert-and-apply updates for all‑time counters, and reuses ~70% fewer lines of code.
AI-written summary · about
chapters (13)
- 0:00Intro
- 2:40What to count - Reads
- 5:11What to count - Requirements
- 6:21How we did it in the past...
- 7:17Drawbacks of the original solution
- 10:13What we want to achieve
- 11:08Simple Counting
- 13:31"Statefuller" Counting
- 15:24Idempotent counting
- 18:42Back to our graphs...
- 20:09All time counters - 1st idea
- 23:16Stats processing pipeline
- 24:48Batch World
related talks
18mReal-time Experiment Analytics at Pinterest with Apache Flink
43mA Data Streaming Architecture with Apache Flink #bbuzz
34mA journey of scaling high-volume low-latency stateful processing
45mApache Flink — Real-Time Data Processing
36mReal Time Big Data Analytics with Kafka, Storm, HBase #bbuzz
36mJoining Infinity: Windowless Stream Processing with Flink
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
27mIntroduction to Flink in 30 minutes