GenAI Training In Production: Software, Hardware & Network Considerations
summary
Meta builds two 24K clusters (one RoCE-optimized, one InfiniBand) and relies on PyTorch to move research to production, mapping tensor/pipeline/data parallelism to server/3K-cluster/cluster layers to match bandwidth and latency. Llama-scale training jumps from ~2T to ~15T tokens and needs thousands of GPUs; a single thermally-throttled GPU causes a 50% performance drop on a 70B run. Solutions include custom collective algorithms, topology-aware schedulers, chassis redesign for higher power/HBM, and operational tooling that reaches ~95% utilization on a rolling 24-hour basis.
AI-written summary · about
chapters (12)
- 0:00GenAI training at Meta
- 1:31Scale shifts with GenAI models
- 4:25Hardware reliability challenges
- 5:30Hardware engineering for GenAI
- 10:11Operational efficiency & failures
- 11:58Failure types & mitigation
- 13:26Efficiency metrics & improvements
- 13:47Network for GenAI clusters
- 16:1524K GPU cluster with RoCE
- 17:41Network performance optimization
- 19:26Parallelism & network topology
- 20:32Summary & future directions
related talks
22mPerformance Optimizations at 100K+ Scale
25mSponsored Session: Accelerating GenAI Inference: From AWS Deep Learning...
12mKeynote: AI Infrastructure
21mDC Networks for Generative AI
34mScaling AI Like a Pro-PepsiCo’s LLM Deployment Strategy on K...
31mBuilding LLaMA: Meta's Director of GenAI
28mTraining Large Language Models on Kubernetes
28mScaling agentic AI: From infrastructure to enterprise outcomes