Keep it going — How to reliably and efficiently operate Apache Flink
summary
A concrete capacity example shows 1 million msgs/s at 2 KB each feeding a five‑minute sliding window with 500 million unique keys, producing ~2 GB/s total and ~400 MB/s per machine on five 10 GbE nodes; using RocksDB with async incremental checkpoints yields ~100 million keys and ~20 GB state per task manager, requiring ~330 MB/s checkpoint throughput (1‑minute interval) and ~760 MB/s read/write bursts — about 50% of network — so tune heap cutoff ratio, slot/CPU mapping and operator chaining plus explicit UIDs and use HA (ZooKeeper + durable snapshot store).
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:34Resource planning
- 16:07Deployment options
- 17:48State backends
- 26:22CPU and memory tuning
- 34:04Checkpoints
- 35:49Common issues
- 37:10High availability
- 45:14Q&A
related talks
42mStream Processing with Apache Flink
50mPanel: Large Scale Streaming in Production
41mWhat If We've Been Scaling Stream Processing Wrong All Along
37mStateful Stream Processing with Apache Flink 1.5 and beyond
20mFlink powers Houzz near-realtime pipeline & Kubernetes logging observability
38mBuilding Unified Streaming Platform at Uber
20mTiny Flink — Minimizing the memory footprint of Apache Flink
31mLogging at Scale, Billion+ messages, 100K msg/sec