The Seven Deadly Sins of Streaming

on YouTube ↗40m

" summary

Claim: Fluss columnar streaming storage replaces Kafka+Flink patterns by exposing streaming tables that are directly queryable and enforce schemas. Demonstrated numbers: an adopter cuts state from 1 TB → 350 GB, checkpoint time 90s → 1s, 85% lower CPU/memory, and Elcommerce user reports eliminating 100 TB of state; cluster handles 40 GB/s and 500k lookups/s. Techniques used: Apache Flink, Apache Arrow, Iceberg/Paimon/Lance formats, column pruning, partition pruning, predicate pushdown; goal is unified hot tables + lakehouse tiering with real-time joins and delta merges.

AI-written summary · about

" chapters (12)

1. 0:00Seven deadly streaming sins
2. 3:55Streaming data pipelines today
3. 9:58Kafka-centric limits
4. 12:10Queryable streaming tables
5. 14:57Change log challenge
6. 15:39Logs vs caches for consistency
7. 18:11Columnar streaming cost cuts
8. 21:00Stateful join bottlenecks
9. 26:15Streaming aggregations
10. 27:37Streaming meets lakehouse
11. 30:51Roadmap and ecosystem
12. 32:56Q&A

" related talks

• 01h 01mApache Fluss: A Streaming Storage for Real-Time LakehouseJark Wu · Future Data Systems Seminar Series (Fall 2025)
• 23mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQLCaito Scherr · Berlin Buzzwords 2021
• 42mStream Processing with Apache FlinkTugdual Grall · DevFest Toulouse 2017
• 41mStream Processor as a Database: Building Online ApplicationsStephan Ewen · Berlin Buzzwords 2016
• 48mUnlocking Real-time Insights: Enhancing Your Databases With Stream ProcessingYingjun Wu · Big Data Conference Europe 2023
• 51mStream Processing Smackdown: Kafka Streams vs. Apache FlinkViktor Gamov · Voxxed Days Ticino 2025
• 43mThe Stream Processor as a DatabaseJamie Grier · Flink Forward Berlin 2016
• 43mUnified stream and batch processing with Apache Flink’s relational APIsFabian Hueske · Flink Forward Berlin 2017