How to configure your streaming jobs like a pro
summary
Keep configuration minimal and practical: set max parallelism = highly composite numbers (120, 720) so key groups divide evenly; estimate parallelism from input rate with stateless ~1M rec/s per task and iterate; trade-offs: use heap backend for low-latency/small state, RocksDB needs SSD and native-memory tuning and creates one RocksDB instance per operator subtask; enable local recovery and snapshot compression, put per-pipeline checkpoint dirs, tune buffer timeout (e.g. 10 ms) for low-rate latency, and automate Kerberos/TLS via the cluster manager.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:49Resource configuration
- 10:03YARN deployment tips
- 12:56Cloud Manager integration
- 19:13Checkpointing explained
- 26:45RocksDB memory tuning
- 31:41Security setup
- 40:19Q&A and summary
related talks
27mDesigning Your Pipelines for Scalability and Resiliency
38mTempering our Production Practices
47mDeploying Like a Boss: From 2 Servers to 20,000
50mPerformance: Avoiding Pitfalls, Diagnosing Issues & Scaling for Growth
46mKeep it going — How to reliably and efficiently operate Apache Flink
22mKeep Ops Happy, Designing for Production
18mScaling, what you need to know
30m8 Tips for Marathon Performance