Building High-Performance, Sustainable AI Clusters
summary
Claims a jump from Titan (2012) to GB200 systems (2024) with roughly 800× energy efficiency and a cost drop from $130M to $400k for comparable work, cites a 1.8T-model training that needed ~15 MW and 8,000 GPUs in 2023, and notes a single large-model checkpoint is ~13.8 TB per checkpoint. Recommends GPU clusters plus fast storage, InfiniBand/NVLink, Base Command/Run:ai for utilization (often <50% without them), offers containerized racks and an immersion-cooled Sustainable Metal Cloud prototype for Europe.
AI-written summary · about
chapters (19)
- 0:00High-performance AI clusters intro
- 1:54AI computing power evolution & Nvidia’s role
- 3:45Generative AI adoption & market
- 5:11Sustainability in AI hardware
- 6:35Why accelerated computing beats CPUs
- 7:41AI infrastructure necessity
- 8:23From prototypes to production AI
- 9:46Key components of AI factory
- 11:52Liquid-cooled data centers
- 13:17SuperPOD for AI workloads
- 13:58Hardware vs software in AI clusters
- 15:46Nvidia’s AI software stack
- 17:10Base Command Manager
- 18:35Cloud bursting & optimization
- 20:39Cost allocation in AI clusters
- 21:19Storage for AI training
- 22:23AI model scaling trends
- 23:26Liquid immersion cooling
- 24:11Summary & Q&A
related talks
24mFIRESIDE CHAT: The Future of AI Infra: The Opportunities and Challenges That Await Us On Our Journey
19mBuilding at Scale with H100: Eos as a DGX SuperPOD Reference Model for Large Data Center Builds
36mTransformers in SRE Land: Evolving to Manage AI Infrastructure
18mTrends Across the AI Frontier
32mFrom Intelligence to Autonomy - Building the Infrastructure for the AI Workforce
21mOperationalizing ML Training Infra at Meta Scale
35mUtilization Is the Key to Efficiency: What It Takes to Run...
20mHacking the Inference Pareto Frontier