Sponsored Session: Fault-Tolerant Training: How We Build Rel...
summary
Nebius describes concrete fault‑tolerance choices and measured results: multi‑stage acceptance tests and continuous active and passive health checks (DCGM, topology, driver checks, tiny GPU fry workloads); virtualized CPU with GPU/infiniband passthrough plus standby nodes for fast replacement; a Slurm-on-Kubernetes operator called soperator (Slurm operator for Kubernetes) and observability across NICs, switches and jobs; tradeoffs between checkpoint frequency and goodput; reported MTBF ≈ 160,000 GPU hours and mean time to recovery ≈ 10 minutes.
AI-written summary · about
chapters (9)
- 0:00Fault-tolerant GPU training
- 1:47GPU cluster failures at scale
- 3:55Measuring GPU cluster reliability
- 7:05Nebius failure reduction
- 10:36Resiliency across the stack
- 14:50Fault-tolerant training frameworks
- 16:18Observability for reliability
- 18:02Nebius reliability achievements
- 19:13Q&A: training vs inference resilience
related talks
21mExperience Operating Large GPU Clusters at Organizational Scale
26mTowards a Fault-Resilient Cloud Management Stack
38mBeyond Kubernetes: Adapting To Spe...
48mFault Tolerance on the Cheap Making Systems That Probably Won't Fall Over
27mFrom Supercomputing to Serving: A Case Study Delivering Cloud Native Foundation Models
42mArchitecting for Reliability
06mKubernetes Challenges at Lambda
47mCloud Native and Sustainable, Reproducible Scientific Computing