Introducing WinBro for Scalable Streaming Graph Partitioning
summary
WinBro implements the first Flink-native, scalable online graph partitioner that runs DBH and HDRF by combining windows and broadcasts to share degree state across parallel tasks. It processes real social graphs including billion-edge Twitter and Friendster datasets, uses millisecond-sized windows to avoid backpressure, and applies periodic state eviction to limit memory. Results show DBH yields higher throughput than HDRF and hash for data-locality-sensitive apps (connected components, triangle count), and the code is available on GitHub.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:14Outline
- 0:55Graph Partitioning
- 2:44Power-Law Graphs
- 3:48HDRF / DBH State Requirements
- 5:50WinBro Job Dataflow
- 6:26WinBro Sample Run
- 10:28Code
- 11:13WinBro State Management
- 12:11Validation & Experiments
- 12:44Streaming Applications (0)
- 14:27Conclusion
- 14:45Special Mention
related talks
38mFaster and Furiouser: Flink Drift
19mDisco: Efficient Distributed Window Aggregation
42mApache Flink deep-dive #bbuzz
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
32mBig Data Processing : Where do we go next ?
37mBuilding and operating large-scale streaming applications at King
22mJoining Dozens of Data Streams in Distributed Stream Processing Systems
34mGoing deep with Spark Streaming #bbuzz