Reliably Sending Events to Apache Kafka: From CDC to the Transactional Outbox Pattern

on YouTube ↗47m

" summary

Covers reliable event delivery to Apache Kafka from relational databases, comparing approaches from CDC (Change Data Capture) to the Transactional Outbox pattern. Describes technologies and trade-offs for guaranteeing delivery and preventing event loss in microservices and distributed systems, and gives guidance for choosing the optimal solution in different situations.

AI-written summary · about

" chapters (18)

1. 0:00Introduction
2. 1:43Distributed systems problems
3. 3:15Kafka delivery guarantees
4. 5:21PostgreSQL vacuum
5. 7:07Synchronous sending issues
6. 8:30Consistency solutions
7. 10:28CDC explained
8. 14:01Transactional Outbox Pattern
9. 16:32PostgreSQL partitions
10. 17:58Lock-based data fetch
11. 20:23Handling stuck messages
12. 21:42Using counters
13. 24:52Kafka event buffering
14. 28:01Transactions and callbacks
15. 28:43Scaling the solution
16. 30:32When to use Kafka Connect
17. 33:09Summary
18. 34:19Q&A

" related talks

• 30mChange Data Streaming Patterns in Distributed SystemsGunnar Morling, Hans-Peter Grahsl · Berlin Buzzwords 2021
• 41mChange Data Streaming Patterns in Distributed SystemsGunnar Morling, Hans-Peter Grahsl · Flink Forward Global 2021
• 45mReliable Toolkit for Event-Driven Microservice ArchitectureYurii Barabash · Software Architecture fwdays'24
• 34mAtomicity and Consistency Across Databases and Messaging SystemsAndrei Shakirin · Devoxx Belgium 2024
• 18mGuaranteed messages delivery:Challenges of transactional Outbox & Inbox patternsAleksandr Gromov · Voxxed Days Thessaloniki 2024
• 42mCreating event-driven microservices: the why, how and whatKatherine Stanley · Voxxed Days Microservices 2019
• 36mChange Data Streaming Patterns for Distributed SystemsGunnar Morling · code.talks 2022
• 21mHow Apache Kafka® Generates Value In Retail With ConfluentAnna Semjen · Big Data LDN 2022