Demystifying Flink Memory Allocation and tuning

on YouTube ↗23m

" summary

Explains the different Flink memory categories (task manager, network, JVM heaps, etc.), shows the calculations used to determine how much memory is assigned to each category, and gives tuning guidance for configuring those memory regions based on specific use cases.

AI-written summary · about

" chapters (10)

1. 0:00Intro
2. 0:11important to know how much data can be stored in
3. 1:50TaskMgr Container Memory Layout
4. 3:27"Cut Off" Space
5. 4:51On or Outside JVM Heap
6. 5:50Configs & Formulas
7. 8:29Hints to Simplify Calculations
8. 11:58Cheat Sheet - Memory/FS state backend
9. 13:24Cheat Sheet - RocksDB state backend
10. 16:07Validating with TM Metrics

" related talks

• 20mTiny Flink — Minimizing the memory footprint of Apache FlinkRobert Metzger · Berlin Buzzwords 2023
• 41mDeep Dive: Spark Memory ManagementAndrew Or · Data By the Bay 2016
• 41mRunning Apache Flink everywhereStephan Ewen · Flink Forward Berlin 2016
• 43mTo Petascale and Beyond: Apache Flink in the CloudsGreg Hogan · Flink Forward Berlin 2016
• 41mIntroducing Flink on MesosEron Wright · Flink Forward Berlin 2016
• 42mApache Flink deep-dive #bbuzzStephan Ewen · Berlin Buzzwords 2015
• 27mData enrichment patterns with Apache FlinkSubham Rakshit · Community Over Code EU 2024
• 27mIntroduction to Flink in 30 minutesDavid Anderson, Johannes Moser · Flink Forward Global 2021