What If We've Been Scaling Stream Processing Wrong All Along

on YouTube ↗41m

" summary

Argues that default horizontal distribution is often needless and builds a single-node alternative preserving Kafka Streams DSL, implementing Flink-like exactly-once via epoch barriers and Kafka transactions, running topology on Java virtual threads from Project Loom and RocksDB state stores. Removes repartition topics and serialization/network hops. On an 8‑core VM it hits ~124k ops/sec with 1KB records (up to ~2M ops for tiny payloads), shows up to 13× lower P99 tail latency, 3.5–4× less CPU and up to 9× less memory, with faster restores; limits are single‑machine caps and single‑producer epoch bottlenecks.

AI-written summary · about

" chapters (7)

1. 0:00Introduction and background
2. 2:59Current stream processing challenges
3. 12:36Rethinking scalability defaults
4. 13:40New vertical scaling architecture
5. 25:20Benchmark methodology and results
6. 31:48Real-world scalability limits
7. 32:51Q&A session

" related talks

• 42mStream Processing with Apache FlinkTugdual Grall · DevFest Toulouse 2017
• 35mStreams Must Flow: Fault-tolerant streams processing apps on KubernetesVictor Gamov · Montréal Cloud Native Day 2019
• 46mKeep it going — How to reliably and efficiently operate Apache FlinkRobert Metzger · Flink Forward Berlin 2017
• 42mPrinciples and Patterns for Streaming Data AnalysisGalder Zamarreño · Voxxed Days Ticino 2018
• 40mDeploying Kafka Streams Applications...Gwen Shapira, Matthias Sax · Scale By The Bay 2018
• 37mStateful Stream Processing with Apache Flink 1.5 and beyondNico Kruber · Berlin Buzzwords 2018
• 13mScaling Kafka on KubernetesShalini Katyayani Koney · Conf42 Kube Native 2025
• 32mScaling your Kafka streaming pipeline....Opher Dubrovsky, Ido Nadler · Scale By the Bay 2021