The Cat and Mouse Game of Infra Cost vs. Latency
summary
Modern AI inference shows P50 ~180ms, P95 ~850ms and P99 ~1.5s so tail events dominate sessions; only ~20% of wall time is GPU compute and ~80% is overhead, fleets run 30–50% GPU utilization, and contention makes latency explode. Four cost hotspots: autoscaling overshoot, GPU underutilization, batching tradeoffs, and correlated tails. Practical fixes are hedged requests (≈35% P99 cut at ~3% extra spend), tiered serving (20–40% total cost cut) and feedback-driven autoscaling (22% less overprovisioning).
AI-written summary · about
chapters (5)
- 0:00Introduction and expectations
- 3:15The problem with tail latency
- 8:54Where the money hides
- 18:06Three response patterns
- 28:44Summary and takeaways
related talks
01h 45mRun Frontier AI at Home
19mReal-Time AI: Low Latency Solutions for Interactive Applications
24mCost and Performance Optimization of LLM Inferencing.
23mThe Race to the Bottom: Low Latency in the age of the Transformer
40mOptimize Your AI Cloud Infrastructure: A Hardware Perspective
37mFrom Hours To Milliseconds: Scaling AI Inference 10x With Serverless on Kubernetes
35mCPU Utilization: The Hidden Cost of Running Hot
15mThis or That with Takara AI