Stability in Large Model Training: Practices in Software and Hardware Fault Self-Healing

on YouTube ↗23m

" summary

Covers Ant Group’s operational practices for keeping GPU fleets running during trillion-parameter model training, including end-to-end GPU monitoring (hardware through application), automated self-healing for 10,000+ GPU clusters (fault isolation, kernel-panic recovery, and node reprovisioning), and using failure-pattern analysis for predictive maintenance to meet SLOs of >98% GPU availability and >90% automatic fault isolation.

AI-written summary · about

" related talks

• 24mCluster Management for Large Scale AI and GPUs: Challenges and OpportunitiesClaudia Misale, David Grove · Cloud Native + Kubernetes AI Day Europe 2025
• 25mTraining Through Failures: How Meta Keeps 100k-GPU Jobs Alive With Open-Sou...T. Rice, A. Afzali · KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026
• 36mTransformers in SRE Land: Evolving to Manage AI InfrastructureQian Ding · SREcon25 Americas
• 17mCharacterization of Large Language Model Development in the DatacenterMeng Zhang, Qiaoling Chen, Peng Sun, Tianwei Zhang, Dahua Lin, Xiaolin Wang, Yingwei Luo, Qinghao Hu, Yonggang Wen, Zhisheng Ye, Zerui Wang, Guoteng Wang · 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI '24)
• 15mSAVE: Software-Implemented Fault Tolerance for Model Inference against GPU Memory..Haibo Chen, Bin Xu, Jinyu Gu, Wenxin Zheng · USENIX ATC '25
• 21mEnhancing Runtime Reliability in LLM Training via Fine-Grained ObservabilityLei Zhang · @Scale: Networking 2025
• 19mAnalysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training WorkloadsFan Yang, Wencong Xiao, Shivaram Venkataraman, Myeongjae Jeon, Amar Phanishayee, Junjie Qian · USENIX ATC '19
• 28mComputing Large Foundational ModelsTom Mason · Techsylvania 2023