Life is But a (Data) Stream: Building Quality Data Pipelines
summary
Argues that shifting to a streams-first, shift-left pipeline replaces brittle batch ETL with real-time, reusable streams and concrete tooling: Apache Kafka (used by 80% of the Fortune 100) for immutable topics, Apache Flink for stateful stream processing, and open table formats like Apache Iceberg or Delta for query-optimized landing. Describes a working product, Table Flow in Confluent Cloud, that auto-derives Iceberg/Delta tables from topic schemas (Avro/Protobuf/JSON Schema) to cut duplicated compute, stop schema-drift garbage, and deliver fresher analytics.
AI-written summary · about
chapters (12)
- 0:00Data pipeline challenges
- 1:27Batch ETL pipeline problems
- 3:54Microservices data contracts
- 6:01Operational vs analytical estates
- 7:26Kafka as streaming platform
- 8:30Stream processing with Flink
- 10:18Open table formats
- 11:23Medallion architecture
- 12:47Processing in streaming layer
- 13:29Stream governance contracts
- 14:33Schema management with Table Flow
- 15:15Conclusion real-time insights
related talks
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming
39mUsing Streaming Data Pipelines in the Cloud to Achieve Success
06mJay Kreps interview
47mBuilding Modern Data Streaming Apps with Open Source
44mThe Changing Face of ETL: Event-Driven Architectures for Data Engineers
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
34mTales of Data Architecture Evolution
43mA Data Streaming Architecture with Apache Flink #bbuzz