Load Imbalance

on YouTube ↗09m

" summary

Describes how MCAS used Apache Flink for real‑time anomaly detection and handled skewed key distributions and huge per‑key state. They split a heavy operator into an Aggregator and Writer, introduced configurable sub‑keys to spread a hot key across N aggregators (MapReduce/combiner style), and replicated/synced per‑key state across those aggregators. To avoid long, blocking iterations over large MapState, they implemented a ShardedMapState wrapper that breaks a key’s MapState into multiple MapState shards plus ValueState so iteration can proceed in chunks without blocking the pipeline.

AI-written summary · about

" chapters (9)

1. 0:00Intro
2. 0:19Microsoft Cloud App Security (MCAS)
3. 1:02MCAS Use Case
4. 1:50Imbalance Problem
5. 2:32Chain of Problems
6. 4:04Partitioning Mechanism
7. 4:36High Volume Customer Solution
8. 6:51Huge State Iteration Problem
9. 7:54Sharded State Solution

" related talks

• 43mDynamic Scaling - How Apache Flink adapts to changing workloadsTill Rohrmann · Flink Forward Berlin 2016
• 29mManaging State in Apache FlinkTzu-Li (Gordon) Tai · Flink Forward Berlin 2017
• 27mData enrichment patterns with Apache FlinkSubham Rakshit · Community Over Code EU 2024
• 42mApache Flink Worst PracticesKonstantin Knauf · Flink Forward Europe 2019
• 42mApache Flink deep-dive #bbuzzStephan Ewen · Berlin Buzzwords 2015
• 37mBuilding and operating large-scale streaming applications at KingGyula Fora · Flink Forward Berlin 2017
• 50mStream Processing Smackdown: Kafka Streams vs. Apache FlinkViktor Gamov · Jfokus 2025
• 37mThe Future of Apache FlinkAljoscha Krettek · Flink Forward Berlin 2016