Data Streaming: Which horse to bet on?
summary
Compares Akka Streams, Kafka Streams, Flink, Spark Streaming and Beam and claims there is no silver bullet: low latency vs high throughput is the core tradeoff. Akka and Kafka deliver sub-second latencies for per-record work (good for MQTT/IoT and clickstream), Flink sits in the middle and supports savepoints and rich windowing, while Spark Structured Streaming uses micro-batches and yields the highest throughput but latencies of seconds to tens of seconds. Beam offers portability but inconsistent back-end feature support. Recommends tools by use case and details checkpointing, watermarking and session-window support.
AI-written summary · about
chapters (24)
- 0:00Intro
- 2:20Cases for data streaming
- 4:20Weird examples
- 5:28Fastest analytics wins
- 8:10Streaming
- 9:06Processing types
- 9:53Time
- 12:28Windows
- 14:33Joints
- 16:21aggregations
- 18:17kafka streams
- 19:27flink
- 20:21Spark Streaming
- 21:09Beam
- 23:27Latency
- 24:25throughput
- 25:24Handling lightness
- 26:38Checkpointing
- 29:38Data Integration
- 30:58Latency vs Fruit
- 32:28Feature Matrix
- 35:49Tradeoffs
- 36:14Business Use Cases
- 40:40Conclusion
related talks
45mData Stream Processing Concepts and Implementations
27mNavigating Streaming Infrastructure
53mDefine data streaming architectures & solutions to fit your needs
39mEvaluating Streaming Framework Performance for a Large-Scale...
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
31mHelp I need a stream processor - learning how to chose ...
41mBeyond the Hype: When Apache Flink Solves Real Problems
28mSweet Streams Are Made Of These