Maximizing Utilization for LLM Accelerators
summary
Measures utilization at the 95th-percentile utilization across a week and breaks losses into five concrete categories: a model-dependent ceiling, unmaterialized demand (chips reserved for late or smaller-than-expected launches), N+1 redundancy overhead, prefill/decode ratio mismatch, and worst of all fragmentation loss. Examples: a 100‑GPU deployment needs ~10 QPS to saturate but sees 1 QPS, wasting 90% of capacity. Remedies built: sampled load tests, redline canaries, 10% ramping, sharing idle capacity, LoRA adapters, instrumentation of TPU tokens/sec, automated forecasting, experiments on 1% flags and autoscaling — all reduce stranded accelerators.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:23Capacity planning basics
- 3:07Measuring utilization limits
- 4:49Utilization loss causes
- 5:34Max utilization ceiling
- 7:01Unmet demand allocation
- 8:48Redundancy trade-offs
- 10:10Disaggregation mismatch
- 10:51Fragmentation loss
- 13:01Consolidating models
- 14:04Automating capacity planning
- 16:14Key takeaways and final recommendations
related talks
56mReduce, Reuse, Recycle: Is this Mantra under attack with Proliferation of Machine(…)
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
20mScheduler and Sharding Considerations for Network Efficiency
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
23mObservability for LLMs: Understanding What’s Happening Under the Hood
15mMLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU...
41mBeyond Uptime: Revolutionizing Fintech Reliability
17mCharacterization of Large Language Model Development in the Datacenter