Faster and Furiouser: Flink Drift
summary
Engineers push Apache Flink on a MapR streaming transport against the Yahoo streaming benchmark (impressions → enrich with campaign ID → keyed windowed counts), discover defaults choke performance, and retune cluster layout: move from one large executor to 3 task managers per node with single application threads, raise partitions from ~10 to 300 partitions, expose hidden threading in SSD stacks, and obtain about a 250% improvement in throughput.
AI-written summary · about
chapters (7)
- 0:00Introduction and motivation
- 1:50Why streaming is revolutionary
- 7:56Decoupling systems with streams
- 15:12How to make streaming fast
- 22:17High-performance streaming platform
- 26:16Benchmark tuning results
- 33:05Q&A and final insights
related talks
42mApache Flink deep-dive #bbuzz
47mBring Flink to Large Scale Production
45mApache Flink — Real-Time Data Processing
39mBetter, Faster, Stronger Streaming: Your First Dive into Flink SQL
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
49mscale.bythebay.io: Stephan Ewen, Apache Flink and the Next Wave of Stream Processing Applications
25mData: Moving with the flow
38mApache Flink: A Very Quick Guide