Feeding the ML model with data from the database
summary
Debezium captures database transaction logs and streams row-level changes into Kafka or directly into consumers so models get real-time updates. Example pipelines use Kafka plus Flink where an online clustering model is trained and served from a single slide of code, or run Flink with Debezium without Kafka to skip Kafka entirely. TensorFlow reads Kafka as strings by default but Debezium’s Single Message Transform converts arrays to CSV so TF can parse and cast them back to numeric tensors.
AI-written summary · about
chapters (5)
- 0:00Introduction and motivation
- 3:19What is Debezium
- 8:12Debezium with Kafka
- 11:02TensorFlow integration
- 13:55Key takeaways and conclusion
related talks
30mChange Data Capture With Flink SQL And Debezium
48mStreaming Database Changes with Debezium
52mMachine Learning Data Pipelines
01h 11mChange data capture pipelines with Debezium and Kafka Streams
43mContinuous SQL with Kafka and Flink
57mMachine Learning Data Pipelines
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications
35mBringing Your Data Pipeline into The Machine Learning Era