Stateful Stream Processing with Apache Flink 1.5 and beyond

on YouTube ↗37m

" summary

Apache Flink 1.5 adds a new deployment architecture with a ResourceManager, Dispatcher and per-job JobManagers enabling dynamic resource allocation and native fits for Kubernetes/YARN/Mesos; the network stack gains credit-based flow control to avoid head-of-line blocking and speed checkpoints; state handling adds fast local recovery by keeping local checkpoint copies while also writing to durable storage; broadcast state lets a non-keyed rule stream be replicated to all tasks for live rule updates; Alibaba reports ~470M records/sec at peak.

AI-written summary · about

" chapters (10)

1. 0:00Introduction and speaker background
2. 1:13What is Apache Flink
3. 8:00Apache Flink 1.5 changes overview
4. 10:51New deployment architecture
5. 15:57Broadcast state in Flink
6. 18:49Network stack improvements
7. 24:48State backends and recovery
8. 27:02SQL and client updates
9. 28:04Future of Flink 1.6 and beyond
10. 30:42Q&A session

" related talks

• 42mStream Processing with Apache FlinkTugdual Grall · DevFest Toulouse 2017
• 37mIntegrate Flink with Kubernetes nativelyYang Wang · Flink Forward Virtual 2020
• 20mAutoscaling Apache Flink ApplicationsRobert Metzger · Berlin Buzzwords 2021
• 46mKeep it going — How to reliably and efficiently operate Apache FlinkRobert Metzger · Flink Forward Berlin 2017
• 33mFlink in ContainerlandPatrick Lucas · Flink Forward Berlin 2017
• 42mManaging Apache Flink on Kubernetes - FlinkK8sOperatorAnand Swaminathan · KubeCon + CloudNativeCon North America 2019
• 25mFlink on Karmada: Building Resilient Data Pipelines on Multi-Cluster K8sMichas Szacillo, Wang Li · KubeCon + CloudNativeCon Europe 2025
• 20mFlink powers Houzz near-realtime pipeline & Kubernetes logging observabilityMenglei Sun · Flink Forward Global Virtual 2020