Distributed Real-Time Stream Processing: Why and How
summary
Compares Apache Storm, Trident, Spark Streaming, Samza, Apex and Flink, showing Spark Streaming uses 1s microbatches while Storm and Flink are native streams; Flink uses distributed snapshot barriers and Apex sends markers for efficient recovery. Explains state handling (Samza replays from Kafka, handles tens of GB; Spark treats state as microbatches), delivery guarantees (at-least-once vs exactly-once), typical latencies (seconds for microbatch, hundreds of ms or lower for native) and throughput expectations (100k–1M records/sec per node); recommends pick by state needs and fault-tolerance tradeoffs.
AI-written summary · about
chapters (25)
- 0:00Intro
- 1:34The Data Deluge
- 1:57Distributed Stream Processing
- 3:07Points of Interest
- 5:19Runtime and Programming Model
- 6:05Native Streaming
- 7:08Micro-batching
- 8:38Apache Streaming Landscape
- 13:10System Comparison
- 17:57Fault Tolerance
- 22:43Managing State
- 26:11Counting Words Revisited
- 28:10Performance
- 30:54Project Maturity [Storm & Trident]
- 31:07Project Maturity Spark Streaming
- 31:25Project Maturity (Samza)
- 31:45Project Maturity [Flink]
- 32:02Summary
- 32:44General Guidelines
- 35:16Recommendations [Storm & Trident]
- 36:09Recommendations Spark Streaming
- 36:49Recommendations (Samza)
- 37:42Recommendations (Apex)
- 38:21Recommendations Flink
- 39:05Dataflow and Apache Beam
related talks
21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & Cassandra
34mGoing deep with Spark Streaming #bbuzz
18mThe role Stream Processing plays for an Opti Channel Experience
42mBuilding a Streaming (E-health) Data Pipeline: When and How?
29mBatch and Streaming Processing in the World of Data Engineering and Data Science
39mStreaming doesn’t have to be hard
30mSmart Streams: A Real-time framework for scoring Big and Fast Data
25mSharing what we love: The Apache Flink story at Pinterest