Fairness in Serving Large Language Models
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
Defines fairness as a token-cost function (input vs decode tokens) and implements a scheduler that updates per-client virtual counters at token granularity, called Virtual Token Counter. Proves a 2× tight upper bound on service difference between backlogged clients for work-conserving, non-preemptive schedules. On traces and synthetic tests with a 20K tokens/sec capacity example VTC recovers unused quota (throughput 14k → 20k) and keeps underloaded clients isolated; prediction (±50%) further reduces unfairness. Code available.
AI-written summary · about
chapters (16)
- 0:00Fairness in LLM serving
- 1:08Aggressive user dominance issues
- 2:34Rate limiting limitations
- 3:38Max-min fairness definition
- 5:03Token-based fairness challenges
- 6:26Formal fairness with backlog
- 7:09Measuring service fairness
- 8:14VTC scheduling algorithm
- 9:18Algorithm theoretical guarantees
- 10:47Evaluation setup
- 11:31Real-world results
- 12:13Synthetic workload analysis
- 13:17Throughput vs rate limiting
- 13:59Weighted VTC fairness
- 14:44Output length prediction impact
- 15:27Summary
related talks
14mFastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
31mLoad Management for AI Models - Managing OpenAI Rate Limits with Request Prioritization
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
35mUsing coding agents with open models
28mServing a Frontier Open Model with Ray & vLLM
28mSponsored Session: Arctic Inference: Breaking the Speed-Cost Tradeoff in LLM Serving
24mLLM Inference Optimization From Simple To Sophisticated
43mLLMs in the Wild - Streaming, RAG, and Real Time GenAI at Scale