Rethinking of fault tolerance in Flink: what lies ahead?
summary
Flink adds unaligned checkpoints (code merged in the 1.10 branch, targeting 1.11) that let checkpoint barriers overtake in‑flight buffers so checkpoint duration drops dramatically under backpressure in tests with ~120 parallelism on 6 nodes, at the cost of larger IO and longer recovery. A planned single‑node recovery path plus a change log state backend aim to enable under‑1s failover and sub‑second end‑to‑end exactly‑once, trading CPU, IO and persistent channels for latency.
AI-written summary · about
chapters (8)
- 0:00Introduction to fault tolerance
- 2:53Current checkpointing limitations
- 7:11Unaligned checkpointing approach
- 9:21Single node recovery vision
- 16:06Non-global failure recovery
- 22:13Sub-second end-to-end latencies
- 25:45Failover under one second
- 30:43Future roadmap and summary
related talks
38mFrom Flink 1.3 to 1.4
36mApache Flink in the Cloud-native Era
28mA look at Flink’s internal data structures & algorithms for efficient checkpointing
26m“Hit me, baby, just one time” – Building End-to-End Exactly-Once Apps w/ Flink
28mRundown of Flink's Checkpoints
44mFlink’s New Batch Architecture
44mNews from Flink’s engine room: “Full steam ahead”
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?