Data Contracts In Practice With Debezium and Apache Flink
summary
Log‑based CDC via Debezium plus Apache Flink converts table change streams into stable, evolvable contracts: the outbox pattern gives transactionally safe, consciously shaped events; Postgres can skip the outbox table with PG logical emit_message; Flink ingest via Flink CDC and Flink SQL performs renames, filters, joins and versioned schemas; Flink 1.20’s array_agg and state hydration solve nested aggregates and Postgres toast gaps; the pipeline reliably publishes to Kafka and lets producers evolve schemas with forward‑compatible changes.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:29Change data capture use cases
- 7:28CDC encapsulation concerns
- 12:26Data contracts for streaming
- 15:19Outbox pattern for encapsulation
- 19:52Stream processing vs outbox
- 23:03Flink SQL for contract transforms
- 28:39Schema changes and compatibility
- 33:38Key takeaways and closing thoughts
related talks
41mChange Data Streaming Patterns in Distributed Systems
30mChange Data Streaming Patterns in Distributed Systems
01h 09mOpen-source Change Data Capture With Debezium
42mChange Data Capture and Processing with Flink SQL
43mChange data capture in production with Apache Flink
19mChange data capture with Debezium…and without
23mChange Data Capture: A RabbitMQ and Debezium Approach
49mChange Data Capture Pipelines with Debezium and Kafka Streams