Networking Optimizations for Multi-Node Deep Learning on Kubernetes
summary
Kubernetes cluster runs multi-node Horovod training using SR-IOV, RDMA over Converged Ethernet (RoCE) and GPUDirect with a Multus + device-plugin setup for per-pod NICs; BIOS ACS, PFC/ECN, kernel GPUDirect modules and source‑based routing are required. On ResNet-50 (≈23M parameters) across five boxes / 48 GPUs, unoptimized runs show one NIC stuck at ~20% throughput while the tuned setup sustains >90% per‑card and delivers roughly 2× end-to-end training throughput.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:04Machine Learning
- 1:19Deep Learning
- 1:47The Very Basics of Neural Networks
- 2:41Neural Networks Can Get Complicated
- 3:50Training and Inference
- 4:59Training Challenges
- 6:30Training Scale Out - Data Parallelism
- 7:48Networking For Scale Out Training
- 10:09GPUDirect RDMA Technology
- 15:02Networking Optimizations
- 16:06Priority Flow Control, Explicit Congestion Notification etc
- 18:44Super Connect - switching architecture
- 21:47Source Based Routing to enable 3
- 23:18Performance/scale measurements
- 26:09Summary and Conclusion!
related talks
25mHigh Performance Networking for Distributed DL Training in Production K8s
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
42mPractice of Building AI Training Cluster Based on Kubernetes+RoCEv2
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
34mDemocratizing Machine Learning on Kubernetes [I]
28mA Deep Dive on How To Leverage the NVIDIA GB200 for Ultra-Fast...
27mMake Kubernetes Networking Ready for World Class AI and HPC ...- 39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...