Rundown of Flink's Checkpoints
summary
Explains how Flink implements Chandy‑Lamport–style distributed snapshots, compares HashMapStateBackend vs RocksDB (memory speed vs disk scale), and shows that you can switch backends on restore from a savepoint. Details checkpoint storage: jobmanager memory is unsafe for production, prefer file system checkpoint storage. Diagnoses long checkpoints via start delay and alignment duration metrics and presents two remedies: unaligned checkpoints (enabled manually, larger snapshots) and buffer debloating (1s default target, disabled by default).
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:44Stateful stream processing
- 2:01Fault tolerance
- 3:00Internal vs external state
- 6:02Distributed snapshots
- 7:29Snapshot alignment
- 10:00Configuring checkpointing
- 17:14Trading Runtime Overhead versus Reprocessing Time
- 18:48Troubleshooting scenarios
- 20:38Slow barrier propagation
- 22:11Long alignment/start delay
- 24:05Configuring unaligned snapshots
- 26:57Configuring buffer debloating
related talks
28mA look at Flink’s internal data structures & algorithms for efficient checkpointing
29mManaging State in Apache Flink
31mRethinking of fault tolerance in Flink: what lies ahead?
26m“Hit me, baby, just one time” – Building End-to-End Exactly-Once Apps w/ Flink
24mFlinkNDB : Skyrocketing Stateful Capabilities of Apache Flink
42mApache Flink Worst Practices
33mFlink and Beam: Current State & Roadmap
44mFlink’s New Batch Architecture