Why WideEP Inference Needs Data-Parallel-Aware Scheduling
summary
WideEP fails not from expert cost but because routing ignores where KV state lives. Using vLLM plus LMD and a router extension called llm-d that reads KV-cache locality and per-replica load, requests prefer replicas with warm KV and spare capacity instead of random fans. Implementation uses Nickel/UCX RDMA, Kubernetes gateway API hooks and data-parallel-aware endpoint selection. Empirically this raises prefix-cache hit rates, boosts output-token throughput (big wins before cache fills around 4k–8k concurrency), and cuts KV duplication and latency spikes when scaling DP from single-node (4 GPUs) to large deployments (up to 48+).
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:26LMD framework overview
- 3:55LM vs classic workloads
- 5:19Intelligent scheduling
- 6:22KV cache management
- 7:05Pre-fill decode disaggregation
- 9:57Tuning pre-fill and decode
- 11:02Wide expert parallelism
- 12:25Why mixture of experts models dominate today
- 14:16Data parallel attention
- 15:01Expert parallel optimizations
- 16:04Wide data parallelism benefits
- 17:30MoE concurrency trade-offs
- 18:16Scaling to 96 GPUs
- 18:38Kubernetes challenges
- 21:50Scheduling benefits in YDP
- 24:19Future LMD work
related talks
19mBeyond the Node: Scaling Inference with Cluster-Wide KVCache...
32mScaling Generative AI Inference with llm-d
47mImprove Load Balancing With Machine Learning Techniques Based on the Sched_ext Framework
19mEfficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance...
20mOPER: Optimality-Guided Embedding Table Parallelization for Large-scale...
15mECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native...
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
44mVLLM K/V Caching With Ceph