“Hit me, baby, just one time” – Building End-to-End Exactly-Once Apps w/ Flink
summary
Explains how Flink uses checkpointing and a two-phase commit sink function to coordinate internal state, external transactions and checkpoint barriers so operators pre-commit and then commit external side effects (or abort) in lockstep; shows concrete connector support: Kafka 0.11 transactions and a transactional file/bucket sink, state stored in the state backend, and tradeoffs like visibility latency vs. atomic output. Measured microbenchmarks show a 5–10% throughput drop versus at-least-once Kafka producers and outlines practical failure/recovery behavior and handling of pending transactions.
AI-written summary · about
chapters (9)
- 0:00Understanding data warranties
- 2:41Single node transactions
- 4:20Distributed system challenges
- 6:44Two-phase commit in Flink
- 12:55Commit and rollback logic
- 16:16Implementing the operator
- 18:57Kafka and Flink connectors
- 20:42Summary and conclusions
- 21:44Q&A session
related talks
27mCustomizing exactly-once processing at scale with Flink
31mRethinking of fault tolerance in Flink: what lies ahead?
42mApache Flink Worst Practices
42mApache Flink Worst Practices
24mStateful Functions: Flexible like a Microservice, Consistent like a Monolith
51mStream Processing Smackdown: Kafka Streams vs. Apache Flink
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
43mStream Processing and Applications in the Modern Age