Designing Scalable Networks for Large AI Clusters: Challenges and Key Insights
summary
Microsoft reports moving from 8K–20K HPC jobs to A100/H100 clusters used across hundreds of thousands of GPUs and claims Top500 cloud rank 1 and MLPerf cloud rank 1 (overall ranks 3 and 2). Describes a rail-optimized non-oversubscribed layout, rail-only and multi-plane variants, and a Smart NIC + simple switch + smart CCL approach that routes using CCL hints. Uses Superbench and IB Pulse to find bad links, applies QP and algorithm tuning, NVLink 2D shuffles, and shows reliability under link flaps while reducing bringup time.
AI-written summary · about
chapters (19)
- 0:00Scalable AI network design
- 2:32Scaling beyond traditional HPC
- 3:34Network topology design
- 4:55Validating AI clusters efficiently
- 7:24Communication library tuning
- 8:47Reliability in AI cluster networks
- 9:50Key insights for network design
- 10:11Topology for public and AI clusters
- 12:36Smart routing with feedback
- 13:41Testing reliability under failures
- 15:07Critical server-to-switch links
- 15:30Frameworks for cluster performance
- 16:35SuperBench and IB Pulse
- 17:18Reducing bring-up time with benchmarks
- 17:39Hierarchical algorithms for AI
- 18:21MLPerf tuned configurations
- 19:04Cost-saving rail-only strategies
- 19:44Non-RTT algorithms for scaling
- 20:07Future network topologies and routing
related talks
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
17mMeta’s Network Journey to Enable AI
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
46mWhen AI Meets High Performance Computing
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
31mLive Panel: Scale-Out Networking for AI Supercomputers
05mSponsored Keynote: Towards Clouds of AI Clusters
22mBeyond Single-Cluster Limits: Scaling GPU Workloads Across Kubernetes With…