Autoscaling Flink at Netflix

on YouTube ↗38m

" summary

Autoscaling routers that consume a single Kafka topic are implemented as a separate Mantis job that polls Atlas every minute, uses statistical short-term prediction (linear or quadratic regression) with 3σ spike detection, and maps a computed target throughput to capacity via a performance table. Rescale pauses and a two-minute metric delay force conservative policies and cooldowns. Deployment reduces fleet CPU usage 25–45% and cuts on‑call pages; future work targets eager scale‑ups, downscale optimization, and multi-source topologies.

AI-written summary · about

" chapters (38)

1. 0:00Intro
2. 0:13Problem Definition
3. 0:25Our Pain
4. 1:36Definitions
5. 2:51Solution Space
6. 4:08Limitations
7. 5:21Assumptions
8. 6:27Desirable Characteristics
9. 7:18Approaches
10. 8:31Autoscaling - High Level Steps
11. 9:52Metrics Collection
12. 10:25Pre-Decision Policy
13. 10:52Decide - Scale Up
14. 12:12Decide - Scale Down
15. 13:51Predict Workload
16. 17:17Spike Detection - First Outlier
17. 17:34Spike Detection - Outliers
18. 18:00Spike Detection - Baseline Reset
19. 18:35Compute Target Processing Rate
20. 21:56Cluster Size Lookup - The Performance Table
21. 23:20Cluster Size Lookup - Corner Case
22. 24:02Cluster Size Lookup - Complexities
23. 25:00Calculate Size Scale Up vs Scale Down
24. 25:27Post-Decision Policy
25. 26:31Running In Production
26. 28:42Autoscaling Architecture
27. 29:46Sine Wave
28. 30:10Linear Spikey
29. 30:29Production Router
30. 30:47Fleet Resource Usage
31. 31:28Additional Considerations
32. 32:38Partition Balancing
33. 34:51Outlier Containers
34. 35:44Future Work
35. 35:54Eager Scale Up
36. 36:35Downscale Optimization
37. 37:12Complex DAGS
38. 37:40Acknowledgements

" related talks

• 20mAutoscaling Apache Flink ApplicationsRobert Metzger · Berlin Buzzwords 2021
• 43mDynamic Scaling - How Apache Flink adapts to changing workloadsTill Rohrmann · Flink Forward Berlin 2016
• 37mSpark and Flink Running Scalable in Kubernetes #bbuzzFrank Conrad · Berlin Buzzwords 2018
• 30mDemystifying Deployments: Applications or Clusters, Active and Reactive Scaling - What is it about?David Moravek · Flink Forward Global 2021
• 32mFlink architecture and automation in Intesa Sanpaologiovani l'auspicio, aim raffaele saginaw, andy fenby, lio verdi · Flink Forward Global 2021
• 37mBuilding and operating large-scale streaming applications at KingGyula Fora · Flink Forward Berlin 2017
• 37mStateful Stream Processing with Apache Flink 1.5 and beyondNico Kruber · Berlin Buzzwords 2018
• 19mKeynote: Making and Keeping Netflix Highly AvailableKatharina Probst · MesosCon Europe 2017