How to Manage State in Apache Kafka

on YouTube ↗44m

" summary

Uses Kafka for transport and storage but finds the event log is not a substitute for a state store, so the team cycles through approaches: reading the log from start (unworkable at scale), caching (loses durability), sending full-state messages (durable but causes 9 MB messages and heavy CPU work), and early Kafka Streams tests (rebalance recovery problems). Final solution combines a database-backed golden copy with CDC from DB to publish lightweight triggers; services see up to 40 minutes startup reduced to seconds.

AI-written summary · about

" chapters (16)

1. 0:00Introduction and context
2. 3:03William Hill use case
3. 6:56State in event logs vs databases
4. 9:27Golden copy of state necessity
5. 11:55Abramson’s law of event logs
6. 13:03Event log as state store myth
7. 16:33Problems with caching and metadata
8. 17:15Full state messages approach
9. 21:57Drawbacks of full state messages
10. 22:18Attempt with Kafka Streams
11. 24:25Return to state stores
12. 27:57State store ownership models
13. 30:22Change data capture pattern
14. 33:13Hardware limitations impact
15. 37:52Cultural lessons and ops collaboration
16. 38:54Summary and key takeaways

" related talks

• 29mApache Kafka and the Next 700 Stream Processing SystemsJay Kreps · Strange Loop 2015
• 33mKafka streams at scaleDeepak Goyal · Rootconf 2019
• 17mOvercome By Events - Apache Kafka And Event-oriented ArchitectureJay Kreps · Big Data LDN 2018
• 53mDemystifying KafkaNakul Mishra · JDD 2019
• 31mEvent Streaming with Kafka StreamsViktor Gamov · Scale By the Bay 2020
• 38mRe-architecting Apache Kafka for the CloudGwen Shapira · Craft Conference 2021
• 27mNo More Silos: Integrating Databases and Apache KafkaRobin Moffatt · Berlin Buzzwords 2021
• 13mKeynote – Jay KrepsJay Kreps · PGConf Silicon Valley 2015