Dayu: Fast and Low-interference Data Recovery in Very-large Storage Systems
summary
Paper builds a centralized timeslot-based scheduler that only schedules tasks likely to finish in the slot, combines a greedy source/destination picker with convex-hull / pocketing to cut candidates to ~1/23, and an iterative WSS bandwidth allocator plus straggler handling. Implementation on a 1,000-node cluster recovering 15 TB and simulation on 3,500 nodes shows ~3x faster recovery than Pangu slow, matches Pangu fast speed with far lower foreground interference, and simulation scales toward 225,000 nodes.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:30Goal of Data Recovery Protocol
- 1:13Observed Production System
- 1:38Very-large Scale
- 2:09Tight Time Constraint
- 2:37Imbalanced Resources
- 3:20Dynamic Foreground Traffic
- 3:42Existing Approaches
- 4:15Dayu: Proposed Recovery Protocol
- 4:51Timeslot-based Scheduling: Methodology
- 5:15Timeslot-based Scheduling: Benefit
- 5:33Scheduling in One Timeslot
- 5:59Choose the Source and Destination
- 7:10Tech 1: Bucket Convex Hull Optimization
- 9:24Underemployed Nodes Recognition (0)
- 9:49Allocate Task's Rate (3)
- 11:14Tech3: Iterative WSS (3)
- 11:36Re-scheduling Stragglers
- 13:03Implementation
- 13:16Evaluation 1: Performance
- 14:42Evaluation 2: Scalability
- 15:23Conclusion
related talks
28mAchieving Rapid Response Times in Large Online Services
16mBoosting Full-Node Repair in Erasure-Coded Storage
16mTowards Optimal Rack-scale μs-level CPU Scheduling through In-Network Workload...
35mMaking Ceph Fast in the Face of Failure
17mDuhu: Shared Disaggregated Memory for Distributed Data Processing Frameworks
01h 12mRedundancy does not imply fault tolerance
27mBuilding Ultra-Resilient Systems in Go
25mAvoiding Overload in Distributed Systems