Lightning Talk: Accelerating LLM Training on Cerebras Wafer-Scale...
summary
Cerebras builds a wafer-scale cluster with an 850,000‑core WSE, 80 GB L1 cache, 1.2 Tb optical IO and 28 kW chassis that treats a 1–128 CS2 cluster as a single PyTorch device. Integration uses LTC and Torch-MLIR to lower models via CS torch and CS torch.compile, enabling near‑linear scaling across scale from one to 128 CS2s without model changes. Weight‑streaming moves weights to Swarm‑X memory nodes, supports static and dynamic sparsity with CSR storage, HDF5 single‑tensor checkpointing, and reported speedups in training and accuracy.
AI-written summary · about
chapters (7)
- 0:00Introduction to Cerebras
- 1:14Weight streaming execution
- 2:13Software interface and API
- 4:00Code example and workflow
- 6:14Scaling and performance
- 7:00Leveraging hardware sparsity
- 8:48Challenges and optimizations
related talks
33mAccelerating the LLM Lifecycle on the Cloud
22mResearch SuperCluster(RSC): Accelerating AI Research at Scale
13mFrom Silicon to Scale: Training at Scale
26mEfficiently serving LLMs at scale.
11mLightning Talk: Accelerating Experimentation...
10mLightning Talk: Scaling Recommendation Systems To 2K GPUs and Beyond
12mAI-powered infrastructure:
34mPanel Discussion: Training and Inference at Planet Scale