MinFlow: High-performance and Cost-efficient Data Passing for I/O-intensive Stateful...
summary
MinFlow is a prototype framework that constructs progressively converging multi-level shuffle topologies, partitions the DAG into interleaved bipartite subgraphs to schedule functions for maximal locality, and uses a configuration model to pick the fastest setting. It claims up to 14× shuffle speedup and an 85% cut in job completion time on terasort versus a single-level baseline, and up to an 86% reduction in storage cost versus prior multi-level designs, evaluated on 10 large EC2 nodes with terasort TPC‑DS and wordcount.
AI-written summary · about
chapters (12)
- 0:00MinFlow introduction
- 1:24Data passing challenges
- 2:51Topology scheduling storage
- 4:40Limitations of optimizations
- 6:09MinFlow’s unified framework
- 6:53Topology Optimizer design
- 9:03Dynamic programming selection
- 10:11Function scheduler localization
- 11:40Optimal configuration model
- 13:30Experimental setup evaluation
- 14:15Performance improvements
- 15:00Conclusion future directions
related talks
19mDShuffle: DPU-Optimized Shuffle Framework for Large-scale Data Processing
26mSplit/Merge: System Support for Elastic Execution in Virtual Middleboxes
26mShuffling, Fast and Slow: Scalable Analytics on Serverless Infrastructure
26mFreeFlow: Software-based Virtual RDMA Networking for Containerized Clouds
45mBeyond Machine Boundaries: Experience with Memory Disaggregation
23mSize-aware Sharding For Improving Tail Latencies in In-memory Key-value Stores
16mA Tale of Two Paths: Toward a Hybrid Data Plane for Efficient Far-Memory Applications
34mA journey of scaling high-volume low-latency stateful processing