Research SuperCluster(RSC): Accelerating AI Research at Scale
summary
A custom cluster delivers concrete hardware and measured gains: a homogeneous fabric with about 48,000 InfiniBand links and no oversubscription, servers with eight 80GB A100 GPUs each totaling 16K GPUs, and peak in-place allreduce bandwidth pushed from ~157 to ~211 Gbps so jobs get data fast. Aggregate compute reaches 5 exaflops. A purpose-built storage layer called Airstore plus 10 PB shared flash cuts data stalls. Llama 65B trains on 2,048 GPUs and No Language Left Behind drops epoch time from ~1 month to ~1 week.
AI-written summary · about
chapters (5)
- 0:00Start
- 2:34How to tackle research challenges?
- 9:10Powerful compute
- 14:34Lessons learned Phased approach gives opportunity to learn lessons
- 16:01Challenges in operating a super cluster
related talks
23mNetworking for GenAI Training and Inference Clusters
21mDesigning Scalable Networks for Large AI Clusters: Challenges and Key Insights
13mFrom Silicon to Scale: Training at Scale
27mThe convergence of machine learning and high performance computing
28mScaling AI Infrastructure
36mAn almost perfect match: cloud native and scientific computing
25mHAL: Computer System for Scalable Deep Learning
25mHigh Performance Networking for Distributed DL Training in Production K8s