MAST: Global Scheduling of ML Training across Geo-Distributed Datacenters at Hyperscale
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
MAST is a global scheduler that automatically places ML input tables and training across Meta regions, modeling table placement as a mixed-integer optimization and using a daily slow pass (Tetris) plus a fast regional allocator. It uses temporal decoupling and scope decoupling and runs exhaustive search across candidate regions for each job. Production results show GPU demand/supply imbalance drops from 2.63 to 0.98 and a 98% GPU allocation rate with training-data non-collocation below 0.1%.
AI-written summary · about
chapters (5)
- 0:00Problem identification
- 4:49Slow-path optimization
- 9:11Scope decoupling
- 11:16Exhaustive search approach
- 14:04Results and conclusion
related talks
15mMLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU...
20mScheduler and Sharding Considerations for Network Efficiency
44mScaling ML from 0 to millions of users
17mServing Heterogeneous Machine Learning Models on Multi-GPU Servers...
39mTrain ML models at scale with Amazon SageMaker Training
21mOperationalizing ML Training Infra at Meta Scale
42mImproving Machine Learning Development Reliability
32mOptimizing Machine Learning Training Infrastructure: A Governance Approach