EoS in Kafka: Listen up, I will only say this once!
summary
Kafka 0.11 delivers exactly‑once semantics by combining an idempotent producer (producer ID + sequence numbers) with a transactional protocol: a transaction log topic, two‑phase commit (prepare/markers/complete), and fencing via an incrementing epoch tied to a transactional ID. Consumer isolation (read_committed/read_uncommitted) and a last‑stable‑offset avoid returning undecided messages while preserving Kafka’s zero‑copy fetch. Overhead concentrates at transaction begin/end (tradeoff: throughput vs commit latency); internal stream processing is covered, external sinks/sources need connector coordination and work remains on latency/speculative reads.
AI-written summary · about
chapters (25)
- 0:00Intro
- 1:31KAFKA IN A HURRY
- 3:36KAFKA PRODUCER
- 4:25PRODUCER PROTOCOL
- 5:20KAFKA CONSUMER
- 6:00CONSUMER PROTOCOL
- 6:42PROCESSING MODEL
- 7:33PROCESSING SEMANTICS
- 8:59AT-LEAST-ONCE SEMANTICS
- 14:13SEMANTIC WEAKNESSES
- 15:04IDEMPOTENT PRODUCER
- 16:03CHANDY-LAMPORT SNAPSHOTS
- 18:59KAFKA TRANSACTIONS
- 23:24TRANSACTION LOG AVAILABILITY
- 25:22ZOMBIE FENCING
- 31:20CONSUMER READ ISOLATION
- 32:24BROKER FETCH HANDLING
- 34:21CONSUMER FETCH HANDLING
- 36:28CONCURRENT TRANSACTIONS
- 38:29KAFKA EXACTLY ONCE SEMANTICS
- 38:48PERFORMANCE OBSERVATIONS
- 39:36END-TO-END EOS
- 41:11FUTURE IMPROVEMENTS
- 41:50EOS TEAM
- 42:15APACHE KAFKA
related talks
53mDemystifying Kafka
01h 00mRabbit MQ vs Kafka: When, Which, Where?
41mKafka as a Platform: The Ecosystem from the Ground Up
26m“Hit me, baby, just one time” – Building End-to-End Exactly-Once Apps w/ Flink
39mKafka as a Platform: The Ecosystem from the Ground Up
40mDistributed Commit Log: Application Techniques for Transaction Processing
01h 00mKafka as a Platform: the Ecosystem from the Ground Up
01h 00mKafka as a Platform: the Ecosystem from the Ground Up