Beyond Round-Robin: GPU-Aware Load Balancing for LLM Inference in Kubernetes
summary
loxilb implements an eBPF-native AI gateway that uses XDP/TC and kernel sockmap for zero-copy L4 forwarding and adds L7 features (API-key, token-quota) plus Consistent Hashing with Bounded Loads. it tokenizes in-process with the HuggingFace Rust tokenizer via CGO, computes vLLM-format KV block hashes from live vLLM ZMQ events, and performs KV-exact routing to GPUs holding those blocks. tests show up to 80% wasted GPU cycles avoided and about 1–2× throughput improvement under overload; epsilon cap 0.75 prevents GPU starvation.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:54GPU workload cache locality
- 9:37Proxy inference gateway
- 14:37AI routing 4-step design
- 23:10Test results performance
- 26:40Similar solutions comparison
- 28:27Production demo
- 32:06Q&A
related talks
27mServing PyTorch LLMs at Scale: Disaggregated Inference With Kubernetes and Llm-d
33mllm-d: Kubernetes Native Distributed Inferencing
14mToppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference
22mBeyond Single-Cluster Limits: Scaling GPU Workloads Across Kubernetes With…
28mProduction-Ready LLMs on Kubernetes: Patterns, Pitfalls, and Performa...
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
32mRunning and Fine-tuning Open Source LLMs on Google Kubernetes Engine
38mOptimizing LLM Inference for the Rest of Us