Experience Productionizing and Operating GB200 Cluster
summary
Shows that multi-node NVLink requires scheduler changes: the team deploys a Slurm topology block plugin to treat each NVL72 domain as a rigid unit, then uses a Slurm simulator (5,000-node / 20,000-GPU equivalent, 15,000 jobs over 7 days, accelerated runtime) to tune policies. Simulation finds occupancy within 1% of no-topology when small jobs can spread and recommends segment size 16 for large MOE jobs; NeMoTron 3 trains at scale with async checkpointing and straggler detection.
AI-written summary · about
chapters (11)
- 0:00GB200 cluster operations intro
- 2:08GB200 system architecture
- 4:13Slurm topology plugin
- 7:22GB200 scheduling challenges
- 10:11Slurm simulator findings
- 13:45Production workload optimization
- 14:25GB200 reliability testing
- 16:56Adaptive testing automation
- 18:00NeMo Tron 3 training
- 19:23GB300 scaling plans
- 20:48Conclusion
related talks
28mA Deep Dive on How To Leverage the NVIDIA GB200 for Ultra-Fast...
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
29mOptimizing Training Performance for Large Language Model(LLM) in Kubernetes
25mHigh Performance Networking for Distributed DL Training in Production K8s
21mDesigning Scalable Networks for Large AI Clusters: Challenges and Key Insights
20mScheduler and Sharding Considerations for Network Efficiency
01h 15mTutorial: Build, Operate, and Use a Multi-Tenant AI Cluster Based Entirely on Open Source