Scaling RoCE Networks for AI Training
summary
Meta runs RoCEV2 on commodity Ethernet to support AI training fabrics with tens of thousands of GPUs, using servers with 8 GPUs and 8 NICs and 16 uplinks per ToR to keep the first two hops non‑oversubscribed. Key software and hardware fixes include DCQCN plus careful PFC/ECN tuning, debugging tools for slow receivers caused by PCIe Gen5 pressure, and load‑balancing via flow multiplexing factor 64 and QP pair ID hashing to avoid elephant‑flow hotspots.
AI-written summary · about
chapters (9)
- 0:00Scaling RoCE for AI training
- 2:15AI training requirements
- 6:50Traffic patterns in AI workloads
- 8:36Network design for AI constraints
- 10:00RoCE deployment challenges
- 11:48Load balancing in RoCE
- 16:46Slow receivers and back pressure
- 18:34RoCE challenges and mitigations
- 20:17Summary and future directions
related talks
22mMetaRoCE: Meta’s RDMA Transport
18mRoCE BALBOA: Service-Enhanced RDMA Offload Engine for Data Center SmartNICs
39mScalable Multi-Node AI Workloads in Multi-Tenant AI Clouds U...
16mMetaRoCE: From Spec to NIC to Open Source
20mMetaRoCE: Meta’s RDMA Transport
19mMetaRoCE: From Spec to NIC to Open Source
22mResilient AI Supercomputer Networking using MRC and SRv6
34mNetworking Optimizations for Multi-Node Deep Learning on Kubernetes