Flink and Beam: Current State & Roadmap
summary
Flink’s Beam Runner implements most Beam API features for both batch and streaming, claiming parity with Google Cloud Dataflow on the Beam capability matrix and supporting watermarks and event‑time windowing, side inputs, and stateful processing. The runner embeds Beam transforms into the Flink runtime, integrates with Flink’s checkpointing and savepoints, and uses Flink’s state backends. Demo shows a job that fails after 10s, restarts from the latest checkpoint and a manual savepoint, and resumes the incrementing counts without data loss.
AI-written summary · about
chapters (13)
- 0:00Introduction and overview
- 1:26What is Apache Beam
- 3:36Beam model and Flink integration
- 7:10Flink runner history
- 10:45Flink runner execution modes
- 13:54Runner strengths and capabilities
- 17:07Flink runner roadmap
- 19:20Flink runtime features
- 21:07Demo setup and program
- 23:56Basic demo execution
- 26:28Fault tolerance and checkpoints
- 28:20Safe points and recovery
- 30:27Q&A session
related talks
26mBeam and dataflow: a perfect combination of parallel processing
47mApache Beam - what do I gain?
39mInteractive Session: Beam and Beam on Flink general roadmap discussion and AMA
21mApache Beam (incubating)
32mStream Processing at Lyft with Flink and Beam
19mBuilding Stateful Streaming Pipelines
39mApache Beam and Google Cloud DataFlow
41mApache Beam: Unified Data Processing