ETL on Kafka + Confluent: Problems and Solutions with Go
summary
Describes building an ETL pipeline for product data using Apache Kafka, Confluent platform and kSQL to handle rapidly changing product information at peaks around 9,000 messages/sec; explains the problems encountered in that setup and how they were solved using Go.
AI-written summary · about
chapters (6)
- 0:00ETL problem
- 5:49Kafka Connect solution
- 9:45Data loss problems
- 15:04Database overload solution
- 25:14Scaling future plans
- 28:06Q&A
related talks
29mGo Big with Apache Kafka
01h 00mPutting Apache Kafka to Use
46mKafka Streams at the Big Data Crossroads at Lightning Speed
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
05mPREVIEW: Kafka Connect Best Practices
39mData Streaming? I don't even know her
06mJay Kreps interview
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...