Faster Than Fast: Networking and Communication Optimizations for Llama 3
summary
Meta builds two 24,000‑GPU clusters (AI zones of ~3,000 GPUs each, 1:7 oversubscription at aggregate switches) to train Llama 3 (8B, 70B, 405B) and reports concrete wins: enhanced ECMP with flow‑multiplexing and new collective algorithms plus switch tweaks (priority queueing for CTS/ACK, tuned credit exchange, slightly larger channel buffers) raise all‑gather performance by >10 percentage points, shaving weeks off multi‑month training. For serving, direct data access over NVLink and fused kernels reduce decode latency for small‑message token generation.
AI-written summary · about
chapters (9)
- 0:00Llama 3 networking intro
- 4:56Training performance tuning
- 6:5924K GPU cluster topology
- 9:02Poor network performance causes
- 12:16Collective communication libraries
- 15:27Network latency improvements
- 17:54Serving inference optimizations
- 22:49Future AI infrastructure directions
- 26:24Q&A
related talks
22mScaling Llama4 Training to 100K
22mPerformance Optimizations at 100K+ Scale
21mDC Networks for Generative AI
31mBuilding LLaMA: Meta's Director of GenAI
23mNetworking for GenAI Training and Inference Clusters
31mThe Evolution of Multi-GPU Inference in vLLM
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
35mLocal LLMs on low end hardware - a practical perspective