Llumnix: Dynamic Scheduling for Large Language Model Serving
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
Llumnix implements live migration of per-request KV caches to reschedule running LLM inferences across instances with near-zero downtime, using a distributed global scheduler plus colocated local schedulers and a virtual usage abstraction to turn diverse goals into simple load balancing. Evaluations on LLaMA models over 16 A10 GPUs show up to 5.5× P99 improvements in first-token latency, a 73% reduction in preemption latency penalties, and reported up to 36% cost savings.
AI-written summary · about
chapters (9)
- 0:00Introduction to LLM serving
- 1:22LLM workload characteristics
- 3:24Scheduling challenges
- 6:02Introducing Llumnix
- 8:04Live migration mechanism
- 10:28Scalable architecture
- 12:23Unified scheduling policy
- 13:40Evaluation and results
- 15:03Conclusion and Q&A
related talks
32mDynamic Scheduling for Large Language Model Serving
15mResource Multiplexing in Tuning and Serving Large Language Models
33mllm-d: Kubernetes Native Distributed Inferencing
33mBeyond VLLM: Distributed LLM Inferencing With Llm-d on Kubernetes
36mCloud-Native Model Serving: vLLM's Lifecycle in Kubernetes
40mScalable and Efficient LLM Serving With the VLLM Production Stack
42mStop Treating LLMs Like REST APIs
34mBeyond Static Pods: Dynamic GPU Sharing and Low-Latency Model Switching for LLM ...