ETL is Dead. Long Live Streams with Apache Kafka

on YouTube ↗40m

" summary

Compares traditional ETL pipelines with stream processing using Apache Kafka, explaining why Kafka-based streaming can replace batch ETL. Covers Kafka technologies and patterns for real-time data movement and transformation, and practical approaches to building streaming data pipelines.

AI-written summary · about

" chapters (14)

1. 0:00Start
2. 6:06Data Services Architecture
3. 7:24Kafka Architecture
4. 8:00Scalable Consumption
5. 9:37Logs and Pub/Sub
6. 11:01Benefits of using Apache Kafka
7. 12:02Crusty Cupcakes
8. 13:08Drum Buffer Rope & Theory of Constraints
9. 14:24Castle Cupcakes
10. 15:06Apache NiFi. Clustering Configuration
11. 15:48Apache NiFi Advantages
12. 16:58KSQL Architecture
13. 18:33KSQL for Streaming ETL
14. 19:07Window Types

" related talks

• 40mThe Changing Face Of ETL: Event-driven Architectures for Data EngineersRobin Moffatt · Øredev 2019
• 08mPREVIEW: Kafka Stream Processing for Everyone with KSQLNick Dearden · Kafka Summit San Francisco 2017
• 41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!Robin Moffatt · All Things Open 2020
• 42mBuilding a Streaming (E-health) Data Pipeline: When and How?Zohreh Jafari · PyData Yerevan 2022
• 31mEvent Streaming with Kafka StreamsViktor Gamov · Scale By the Bay 2020
• 29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data PipelineRobin Moffatt · DevOps Pro Europe 2021
• 17mOvercome By Events - Apache Kafka And Event-oriented ArchitectureJay Kreps · Big Data LDN 2018
• 49mLet’s Build a Streaming Data Pipeline!Robin Moffatt · DataXDay 2020