Building a 5-Exaflop Supercomputer for Meta-AI Research and...
summary
A single cluster delivers 5 exaflops of aggregate compute with 16,000 GPUs across three halls, a non‑oversubscribed InfiniBand fabric, roughly 500 PB of capacity and 10 PB of flash for code and checkpoints, all scheduled by SLURM. The team builds a purpose‑built cache called AirStore cache to feed random and iterative samples at 6 TB/s, debugs optics, cable bend and subnet‑manager failures (one outage lasts 2.5 hours), and trains Llama 2; it mostly works but forces vendor fixes and new observability.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:03AI training workloads challenges
- 4:13Large-scale AI training challenges
- 6:45Research Super Cluster design
- 8:51AIRStore caching system
- 11:21InfiniBand fabric role
- 13:51User interaction via scheduler
- 16:25HPC vs Meta infrastructure
- 18:31Vendor collaboration reliability
- 20:19Reliability metrics in HPC
- 23:06Debugging InfiniBand at scale
- 25:11Failure complexity in systems
- 26:37RDMA and remote memory
- 28:24Software bugs in AI workloads
- 29:27Models impact on cluster
- 30:10Navigating uncharted territory
- 31:13Globally distributed RSC team
- 32:16Software and production collaboration
- 33:41Contributing to system reliability
- 34:27Closing
related talks
46mChallenges of Making Large AI Clusters Reliable
02mStarcluster
26mExo: Run your own AI cluster at home
21mBuilding Distributed Platforms at Extreme Scale on Fast Changing Data
23mBreaking the limits of x86
51mSustainable & Smart AI Factories in Europe
20mLegion: Automatically Pushing the Envelope of Multi-GPU System for Billion-Scale...
16m[Opening Keynote] Making Distributed Computing Easy