Demystifying Deployments: Applications or Clusters, Active and Reactive Scaling - What is it about?

on YouTube ↗30m

" summary

Explains concrete deployment choices for Apache Flink: application mode bundles job and cluster to avoid jar distribution, session clusters use a two-step submit requiring task managers with slots (default one slot each) and examples show a job with parallelism 2 needing two slots. Shows container and orchestration integrations—native Kubernetes, Docker, YARN—and demos reactive scheduler mode on Kubernetes that scales from one to three TaskManagers, auto-rescales using the latest checkpoint (restored checkpoint 9). Notes Mesos deprecation and YARN per-job legacy mode.

AI-written summary · about

" chapters (10)

1. 0:00Introduction
2. 1:11Standalone Resource Provider
3. 8:52Application Mode
4. 12:30Docker
5. 14:46Kubernetes
6. 16:33Demo - The Reactive Mode on Kubernetes
7. 20:50Native K8s Session
8. 24:19Native K8s Application
9. 24:56YARN
10. 26:52The deployment decision matrix

" related talks

• 41mRunning Apache Flink everywhereStephan Ewen · Flink Forward Berlin 2016
• 20mAutoscaling Apache Flink ApplicationsRobert Metzger · Berlin Buzzwords 2021
• 41mIntroducing Flink on MesosEron Wright · Flink Forward Berlin 2016
• 43mDynamic Scaling - How Apache Flink adapts to changing workloadsTill Rohrmann · Flink Forward Berlin 2016
• 38mAutoscaling Flink at NetflixTimothy Farkas · Flink Forward Virtual 2020
• 37mBuilding and operating large-scale streaming applications at KingGyula Fora · Flink Forward Berlin 2017
• 42mApache Flink deep-dive #bbuzzStephan Ewen · Berlin Buzzwords 2015
• 42mApache Flink meets Apache Mesos and DC/OSTill Rohrmann · Flink Forward Berlin 2017