Serving LLMs for Agents at Scale with Ray + vLLM
summary
Self-hosted Ray + vLLM platform reduces end-to-end latency from ~2,400 ms to 275 ms and reports about 10x performance and 12x cost versus third‑party hosts, using Ray on EKS, vLLM, H100/B200 experiments, and tenant‑specific Ray clusters. Key engineering work: speculative decoding with a shared suffix tree to lift acceptance by ~35%, tokenizer and KV‑cache tuning, GCS (Redis) fault‑tolerance for headpod survival, and incremental Ray cluster upgrades for zero‑downtime deploys.
AI-written summary · about
chapters (4)
- 0:00Introduction and motivation
- 5:54VLM optimizations overview
- 14:29High availability strategies
- 25:12Key takeaways and Q&A
related talks
11mvLLM & Ray Updates
25mIntelligent Data Classification with Ray and vLLM at Apple
28mAnyscale's Unified Platform for LLM Development and Deployment
30mAccelerated LLM Inference with Anyscale
40mScalable and Efficient LLM Serving With the VLLM Production Stack
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
13mRay and Anyscale: An Optimization Journey
29mHow Roblox Scaled Machine Learning by Leveraging Ray for Efficient Batch Inference