Dynamic Scaling - How Apache Flink adapts to changing workloads

on YouTube ↗43m

" summary

Apache Flink implements dynamic scaling by splitting keyed state into key groups and non-key state into small units so the system can reassign state when changing parallelism; Kafka source offsets become per-partition Kafka offsets so partitions move between subtasks. Scaling requires taking a safe point snapshot, canceling, and restarting with a new parallelism — a demo shows parallelism 1 → 3 stops Kafka lag growth and then reduces it. Current limits: max parallelism (key-group count) is fixed at job start and live operator rescale and stop-with-atomic-snapshot are planned.

AI-written summary · about

" chapters (7)

1. 0:00Introduction to dynamic scaling
2. 3:59Challenges of rescaling Flink jobs
3. 12:49Partitioning state in Flink
4. 17:05Live demo of scaling a job
5. 29:41Current limitations and future plans
6. 33:38Summary and release timeline
7. 34:23Q&A session

" related talks

• 20mAutoscaling Apache Flink ApplicationsRobert Metzger · Berlin Buzzwords 2021
• 38mAutoscaling Flink at NetflixTimothy Farkas · Flink Forward Virtual 2020
• 09mLoad ImbalanceDmitry Voronov · Flink Forward Global Virtual 2020
• 50mStream Processing Smackdown: Kafka Streams vs. Apache FlinkViktor Gamov · Jfokus 2025
• 36mApache Flink in the Cloud-native EraStephan Ewen · Flink Forward Global 2021
• 30mDemystifying Deployments: Applications or Clusters, Active and Reactive Scaling - What is it about?David Moravek · Flink Forward Global 2021
• 44mFlink’s New Batch ArchitectureTill Rohrmann, Ma Guowei · Flink Forward Europe 2019
• 42mApache Flink deep-dive #bbuzzStephan Ewen · Berlin Buzzwords 2015