Training Through Failures: How Meta Keeps 100k-GPU Jobs Alive With Open-Sou...
Talkat KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026 · 2026-09-07..09-09·
summary
Meta reports keeping 100k‑GPU training jobs alive by treating GPU failures as expected and recovering per step in under 30 seconds for many cases. Open-source pieces include fault‑tolerant collectives in PyTorch (torchcomms) with MCCL and NCCL primitives ncclShrink ncclGrow ncclCommRevoke, a peer‑to‑peer checkpointing stack centered on torchstore, and low‑level RDMA tooling (RDMA for Py) that hits line rate and enables NVMe‑over‑Fabric writes around 28 GB/s for fast, durable recovery.
AI-written summary · about
chapters (13)
- 0:00Fault-tolerant training intro
- 1:46Primitives over end-to-end
- 3:31Training failure scenarios
- 5:57PyTorch fault-tolerant collectives
- 9:26NCCL dynamic world size APIs
- 11:53RDMA fundamentals for GPU
- 14:45GPU memory registration APIs
- 15:06RDMA Py low-level wrapper
- 18:57Custom fault-tolerant collectives
- 19:20Checkpointing in systems
- 21:51TorchStore peer recovery
- 23:55NVMe over fabric storage
- 24:18Q&A
related talks
21mOperationalizing ML Training Infra at Meta Scale
20mTaming AI Infrastructure Failures with Agentic Debugging
24mSponsored Session: PyTorch Distributed and Fault Tolerance
23mStability in Large Model Training: Practices in Software and Hardware Fault Self-Healing
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
19mAnalysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
25mHigh Performance Networking for Distributed DL Training in Production K8s
42mImproving Machine Learning Development Reliability