Utilization Is the Key to Efficiency: What It Takes to Run...
summary
Cloudflare runs almost 100 models on GPUs spread across 200+ cities and tackles low utilization with three core components: an edge coordinator, permit-based routing, and a scheduler that forecasts load. A bespoke inference engine uses just-in-time kernels and per-batch CUDA graphs to cut launch cost, yielding 7% throughput vs VLM (up to 50% in CPU-constrained cases). An Omni layer overcommits to ~400% GPU memory per device, swapping via unified memory to save about four GPUs per instance. Random nearby-data-center routing proves best by chance for free-permit discovery.
AI-written summary · about
chapters (18)
- 0:00Cloudflare AI inference intro
- 1:21Scaling AI inference on edge
- 4:59Underutilization and GPU efficiency
- 7:10Edge inference architecture
- 9:16GPU isolation in containers
- 10:42Model placement and memory
- 12:09Bespoke inference engine
- 13:14Multi-model GPU allocation
- 15:44Edge coordinator and distribution
- 18:32Permit manager for fairness
- 20:20Diffusion models and CPU offload
- 21:44Load balancing and KV caching
- 23:12Fair queuing for resources
- 25:45Scheduler's global role
- 27:30Utilization metrics and scaling
- 29:16Model routing optimization
- 30:41Key lessons from platform
- 32:46Constraints driving innovation
related talks
43mThe GPU Utilization Problem: What’s Going Wrong and How to Solve It
16mTowards GPU Utilization Prediction for Cloud Deep Learning
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
41mBeyond Uptime: Revolutionizing Fintech Reliability
33mAI workloads on Kubernetes - How to maximize GPU utilization and cut costs
17mCharacterization of Large Language Model Development in the Datacenter
26mEfficiently serving LLMs at scale.
38mInference in Progress… Please Monitor Responsibly