Behind the scenes scaling ChatGPT
summary
Launch day is November 30 2022 then Japan and an Elon tweet spike traffic the next day exposing a hard GPU cap that shows up as a visible fail whale page. Scaling relies on hardware details: clusters of A100 GPUs with NVLink and HBM, 3.35 TB/s memory moves versus 1.98 quadrillion 8‑bit ops leading to a 591:1 ops to byte ratio. KV‑cache evictions force full recomputes, batch size and cache residency become the real load metrics, teams grow from ~30 to ~100 while juggling multi‑region deployment and active abuse mitigation.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:50GPUs and language models
- 10:30Memory bandwidth and batch size
- 14:04Global GPU supply constraints
- 17:39Growing engineering team
- 21:31AI safety and abuse handling
- 25:24Future scaling challenges
- 26:30Acknowledgments
related talks
28mShared Yet Isolated at Scale: Building Multi-Tenant Inference Platform on…
12mKeynote: AI Infrastructure
29mLive Keynote
17mCharacterization of Large Language Model Development in the Datacenter
47mScaling: from 0 to 25 million users
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
21mMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs
22mGenAI Training In Production: Software, Hardware & Network Considerations