How To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
summary
On‑prem GPU platform uses Envoy AI Gateway and Athenz for auth, serves 10 models to 30 tenants and handles 650B tokens/month with shared model servers to boost GPU utilization. Benchmarks show FB8 quantization beats BF16 on throughput and latency; speculative decoding gives 51% acceptance on shared GPT but only 15% on random prompts; KV cache aware routing cuts TTFD in multi‑turn workloads from ~1,200ms to ~200ms. PD disaggregation improves ITL when PD:decode ratios (example 3:1) match traffic but adds RDMA and orchestration complexity.
AI-written summary · about
chapters (13)
- 0:00LLM inference optimization
- 1:04Platform design & shared servers
- 4:40Workload analysis & use cases
- 7:45Latency vs throughput tradeoffs
- 9:55Agentic workload challenges
- 11:01Optimization evaluation framework
- 12:47Quantization for speed/memory
- 15:15Speculative decoding
- 18:08KV cache routing explained
- 22:28Prefill-decode disaggregation
- 27:43Future workload trends
- 28:48Key takeaways
- 29:30Q&A on routing
related talks
28mNavigating LLM deployment: Tips, tricks, and techniques
12mVertical Mobility: Inference from MVP to Trillion Parameters
26mEfficiently serving LLMs at scale.
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
42mStop Treating LLMs Like REST APIs
31mThe Evolution of Multi-GPU Inference in vLLM
18mServing Fast and Smart: Getting the Most Out of Ray Serve LLM on GKE
15mAI Trends: Insights