The Changing Face Of ETL: Event-driven Architectures for Data Engineers
summary
Argues that modeling everything as events avoids brittle ETL and shows concrete patterns with Apache Kafka: stream‑table duality, Kafka Connect for sources/sinks, Kafka Streams and KSQL for joins/enrichment, Schema Registry and Avro for schemas. Walks through a live example (user reviews → cleanse → enrich with user DB → VIP alert topic) and explains offsets, partitions, log compaction and replays so you see how to build, scale and reprocess pipelines rather than vague theory.
AI-written summary · about
chapters (18)
- 0:00The Changing Face of ETL
- 11:13The Stream/Table Duality Table
- 14:02What is an Event Streaming Platform?
- 14:44Immutable Event Log
- 15:38Topics
- 16:16Partitions
- 16:41Messages are just K/V bytes
- 20:40Streaming Integration with Kafka Connect
- 21:19Stream Processing in Kafka
- 21:44Kafka Streams API
- 23:10Events in Action
- 30:59Not Everything is a Nail
- 31:53Side-by-Side Tech Evaluation
- 32:45Evolve Data Sources
- 33:56Tight Coupling != Flexible
- 35:25Loose Coupling == Freedom to Evolve
- 35:36Transform Once, Use Many: Data Cleansing
- 37:14Message Payload Compatibility
related talks
39mModern Event-driven Architecture: Adopting Data in Motion
40mETL is Dead. Long Live Streams with Apache Kafka
17mOvercome By Events - Apache Kafka And Event-oriented Architecture
43mBack to fundamentals: from databases to event-driven platforms
49mLet’s Build a Streaming Data Pipeline!
58mTales of Data Architecture Evolution
41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
41mChange Data Streaming Patterns in Distributed Systems