Change data capture pipelines with Debezium and Kafka Streams

on YouTube ↗01h 11m

" summary

Shows how Debezium + Kafka Streams capture DB transaction logs and run low‑latency streaming queries (examples claim sub‑2‑second delivery). Demonstrates an implemented pipeline that enriches sensor events with master data (weather stations), builds per‑key aggregates and emoji visualizations in dev mode, and materializes aggregate views while preserving transaction boundaries using a transaction metadata topic. Notes practical limits (Postgres replication‑slot failover can lose events) and running Kafka Streams on Quarkus native via GraalVM.

AI-written summary · about

" chapters (13)

1. 0:00Introduction and agenda
2. 4:00Speaker introduction
3. 6:43Change data capture basics
4. 12:59Event structure and serialization
5. 17:31Postgres failover discussion
6. 20:32Deployment and query-based CDC
7. 24:06CDC and stream processing
8. 27:13Kafka Streams overview
9. 30:38Quarkus framework introduction
10. 35:27Quarkus internals and optimization
11. 40:41Developing with Quarkus
12. 46:59Kafka Streams and Debezium demo
13. 53:46Advanced patterns and conclusion

" related talks

• 03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...F. Ramière, D Petisme · Devoxx France 2021
• 31mChange Data Capture for non relational databasesJakub Cechacek · DevConf.CZ 2023
• 30mChange Data Capture With Flink SQL And DebeziumMarta Paes · DataEngBytes 2020
• 43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · LISA18
• 53mApache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!Viktor Gamov · Jfokus 2019
• 48mStreaming Database Changes with DebeziumGunnar Morling · Devoxx Belgium 2017
• 27mUnderstanding stream table duality using Kafka, Flink SQL and Debezium formatMuhammet Orazov · Community Over Code EU 2024
• 42mBuilding a Data Pipeline with Clojure and KafkaDavid Pick · Clojure/conj 2014