No More Silos: Integrating Databases and Apache Kafka
summary
Argues there are two concrete CDC approaches for getting database changes into Kafka: query-based CDC (JDBC connector) polls a timestamp or incrementing id, is easy to set up, but cannot reliably capture deletes or every intermediate event and increases DB load; log-based CDC (Debezium or Confluent log connectors) reads the transaction log, captures every insert/update/delete with before/after state, gives event-level fidelity and low source impact but requires extra permissions and setup. Recommends Kafka Connect as the integration platform.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:36Why get data from your database into Kafka
- 4:12Stream table duality
- 5:32The log
- 8:05Change data capture
- 10:42Querybased change data capture
- 12:15Creatorbased change data capture
- 13:55Createbased order data capture
- 17:23Logbased change data capture
- 18:48Logbase change data capture
related talks
43mLoading into Kafka from a Database: With and Without CDC
43mBack to fundamentals: from databases to event-driven platforms
01h 09mOpen-source Change Data Capture With Debezium
48mStreaming Database Changes with Debezium
38mCDC Without Pain: real-world resilient replication with Debezium and Kafka
41mChange Data Streaming Patterns in Distributed Systems
39mDeep Dive on Using YugabyteDB's Change Data Capture Connector
44mHow to Manage State in Apache Kafka