From Zero to Hero with Kafka Connect
summary
Live demo shows a MySQL orders table captured into Kafka via MySQL source connector, every insert update and delete becomes a Kafka event, then the same topic fans out to Elasticsearch and Neo4j sinks to enable real-time search and graph analytics; connectors run as JVM workers with tasks and store offsets/config in Kafka so clusters are fault tolerant; advice is to use Avro + Schema Registry for schema governance; demo runs successfully and handles transformations, retries and dead‑letter routing.
AI-written summary · about
chapters (24)
- 0:00Start
- 1:54Streaming Integration with Kafka Connect
- 3:19Streaming Pipelines
- 4:42Writing to data stores from Kafka
- 17:36Kafka Connect basics
- 20:48Serialisation & Schemas
- 22:13The Confluent Schema Registry
- 25:05Single Message Transforms
- 28:28Extensible
- 30:15Connectors and Tasks
- 31:23Tasks and Workers
- 32:46Kafka Connect Standalone Worker
- 33:31"Scaling" the Standalone Worker
- 34:14Kafka Connect Distributed Worker
- 36:08Distributed Worker - fault tolerance
- 36:23Multiple Distributed Clusters
- 37:07Kafka Connect images on Docker Hub
- 37:22At runtime
- 38:16Troubleshooting Kafka Connect
- 39:59Mismatched converters
- 40:16Mixed serialisation methods
- 40:39Error Handling and DLQ
- 42:06Re-processing the Dead Letter Queue
- 42:20REST API
related talks
42mFrom zero to hero with Apache Kafka's connect API
39mKafka as a Platform: The Ecosystem from the Ground Up
43mApache Kafka and KSQL in Action: Let’s Build a Streaming Data Pipeline!
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
44mNo More Silos: Integrating Databases and Apache Kafka
01h 07mKafka Streams and KSQLDB
41mKafka as a Platform: the Ecosystem from the Ground Up
50mIntroduction to Apache Kafka