Build a data pipeline in 20 minutes with Kafka Connect

on YouTube ↗18m

" summary

Shows how to build a Kafka Connect data pipeline in 20 minutes: use Debezium CDC to capture MySQL change events, send them to Kafka topics (with before/after fields and auto-created topics), and archive data to an S3 sink partitioned by year/month/day. Configuration is done via the Connect REST API, workers rebalance tasks across the cluster, files are written in batched uploads, and archived data is queried with Amazon Athena for joins, aggregation and exploration.

AI-written summary · about

" chapters (16)

1. 0:00Introduction
2. 1:52Kafka overview
3. 3:56Kafka Connect basics
4. 6:23Connect architecture
5. 7:06Use case demo
6. 8:08Docker setup
7. 8:49Microservice data production
8. 9:10Debezium for CDC
9. 10:34Verify Debezium data
10. 11:15Real-time demo
11. 12:21Kafka to S3 sink
12. 13:44Check S3 files
13. 14:45Athena analysis
14. 16:09SQL on archived data
15. 17:10Cross-referencing reports
16. 17:31Conclusion

" related talks

• 49mLet’s Build a Streaming Data Pipeline!Robin Moffatt · DataXDay 2020
• 41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · All Things Open 2020
• 29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data PipelineRobin Moffatt · DevOps Pro Europe 2021
• 49mChange Data Capture Pipelines with Debezium and Kafka StreamsGunnar Morling · Developer Summit 2020
• 43mLoading into Kafka from a Database: With and Without CDCAndrey Serebryanskiy · JPoint 2023
• 42mData Streaming for Microservices using DebeziumGunnar Morling · Voxxed Days Microservices 2018
• 01h 09mOpen-source Change Data Capture With DebeziumGunnar Morling · Vaccination Database Tech Talks (Booster) - Spring 2022
• 35mApache Kafka as a Monitoring Data PipelineJakub Scholz · DevConf.CZ 2021