From zero to hero with Apache Kafka's connect API

on YouTube ↗42m

" summary

Live demo wires a MySQL binlog source into Kafka via Kafka Connect and pushes the same topic to two sinks: Elasticsearch and Neo4j, using Avro serialization and a schema registry. Configuration shows connector vs task vs worker, standalone vs distributed modes, and simple Single Message Transforms. Troubleshooting covers the common “magic byte” Avro vs JSON error, dead‑letter topics, and offset recovery; the pipeline mostly runs successfully despite a few gremlins.

AI-written summary · about

" chapters (7)

1. 0:00Introduction to Kafka Connect
2. 5:06Live Kafka Connect demo
3. 12:29Understanding Kafka Connect internals
4. 22:12Deploying Kafka Connect
5. 29:03Troubleshooting Kafka Connect
6. 36:28Running Kafka Connect in containers
7. 37:54Q&A and closing

" related talks

• 45mFrom Zero to Hero with Kafka ConnectRobin Moffatt · Voxxed Days Microservices 2019
• 44mNo More Silos: Integrating Databases and Apache KafkaRobin Moffatt · code.talks 2019
• 39mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia Europe 2020
• 50mIntroduction to Apache KafkaJames Ward · Devoxx US 2017
• 43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · LISA18
• 53mApache Kafka and KSQL in Action : Let’s Build a Streaming Data Pipeline!Viktor Gamov · Jfokus 2019
• 05mPREVIEW: Kafka Connect Best PracticesRandall Hauch · Kafka Summit San Francisco 2017
• 26mConnecting distributed data to Azure CosmosDB with Apache KafkaJacob Bogie, TJ Laher · Azure Cosmos DB Conf 2022