Automatic, Application-Aware I/O Forwarding Resource Allocation
summary
A system called DFRA uses low-overhead, history-based I/O monitoring (BEACON) to decide at job dispatch whether to give a job extra forwarding nodes or dedicated forwarding nodes; decision rules check per-job volume, node profiles and a calculated required forwarding capacity. Deployed on Sunway TaihuLight, remapping completes in seconds, removes most pairwise interference (some workloads hit up to 16.0x I/O speedup), and saved ~200 million core-hours in eight months while operating opt-in by job submission.
AI-written summary · about
chapters (12)
- 0:00Intro
- 0:36Storage Crucial for Supercomputing
- 3:54Sample Forwarding Configuration: TaihuLight
- 6:09Resource Misallocation
- 10:28Forwarding Node Anomaly
- 13:31Automatic Forwarding Node Scaling
- 19:00DFRA Workflow
- 19:53Implementation and Deployment Status
- 23:25Upgrade Eligibility of Historical Jobs
- 24:56I/O Interference Reduction
- 26:02Remapping Overhead vs. Saving
- 26:53Conclusion
related talks
25mEnd-to-end I/O Monitoring on a Leading Supercomputer
20mFull Lifecycle Data Analysis on a Large-scale and Leadership Supercomputer...
43mNo shard left behind: Dynamic work rebalancing in Apache Beam
45mUber achieves millions $ of savings managing disk IO across HDFS cluster
16mAI4DL: Mining Behaviors of Deep Learning Workloads for Resource Management
38mFaster and Furiouser: Flink Drift
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed...
22mData transport in a radio telescope: Remote Direct Memory Access over Ethernet from FPGA to GPU