Networking for GenAI Training and Inference Clusters
summary
Large language model workloads drive orders of magnitude more compute and new network designs: training examples cite a 70B-parameter Llama2 trained on ~2 trillion tokens requiring roughly 1.7 billion GPU hours and prompting plans for 32k GPUs to reach ~30 exaFLOPS usable. Distributed inference splits into pre-fill (parallel, compute heavy) and decode (memory bound, per-token latency) with targets like time to first token <1s and ~50 ms per incremental token. Rocki/RoCE and InfiniBand fabrics both appear in production, and model parallelism forces cross-node scale-up bandwidth and lower-latency fabrics.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:59Use case: Generative Al - Content generation
- 3:37Recommendation models vs LLMs
- 5:02Recent LLM models from Meta
- 8:36Generative Al: text generation with Large Language Models (LLM)
- 9:24LLM Inference: dual-edge problem
- 10:44Recap slide & Transition
- 11:53The compute fabric for LLM training
- 13:41A training cluster network primer: 32K scale
- 15:18Traffic patterns from collectives
- 17:19Mapping collectives onto the network
- 18:38LLM training fabrics: the challenges
- 20:28LLM inference needs a fabric too!
related talks
22mResearch SuperCluster(RSC): Accelerating AI Research at Scale
06mOpening Remarks
27mFaster Than Fast: Networking and Communication Optimizations for Llama 3
21mDC Networks for Generative AI
22mPerformance Optimizations at 100K+ Scale
15mAlibaba HPN: A Data Center Network for Large Language Model Training
19mAI/ML Networking Challenges: The Fast and the Finicky!
21mDesigning Scalable Networks for Large AI Clusters: Challenges and Key Insights