Serving Fast and Smart: Getting the Most Out of Ray Serve LLM on GKE
summary
Ray Serve LLM on GKE replaces the Python proxy with HAProxy and splits control and data planes via an ingress request router so streaming tokens go directly to model replicas, removing the per-token hop. Benchmarks on GEMA 4 E2B across 8 B200 GPUs report about 9x streaming throughput gains, 4.4x prefill and 24x decode-heavy throughput improvements, and a validation case showing ~5x better scaling with ~8x lower latency, integrated with Cubray and Inference Gateway.
AI-written summary · about
chapters (14)
- 0:00Introduction and speakers
- 0:50Ray Serve and LLM overview
- 2:13Scaling bottlenecks identified
- 4:22Proxy upgrade with Haroxy
- 6:30Control and data plane separation
- 8:17Performance gains with optimizations
- 9:20Ray Serve LM API benefits
- 10:23GKE validation and benchmarks
- 11:47Inference Gateway on GKE
- 13:54Multi-cluster routing setup
- 15:39Single Ray Service routing
- 15:59Ray Service declarative API
- 16:40TPU support on GKE
- 17:22GKE and Ray serving recap
related talks
31mSpotify's Approach to Distributed LLM Training with Ray on GKE
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
27mYes You Can Run LLMs on Kubernetes
21mOptimizing Model Serving on Kubernetes With Model Streaming
31mHosting AI models with Ray and GKE
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
20mInferencing LLMs in production with Kubernetes and KubeFlow
28mScaling AI & Machine Learning Workloads With Ray on AWS, Kubernetes, & BERT