Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
Describes a scheduler that splits large prefill passes into chunked-prefills and constructs stall-free schedules that mix those chunks with decode batches to avoid stalls while keeping GPUs saturated. Uses arithmetic-intensity analysis to choose chunk sizes, supports pipeline parallelism, and releases code at github.com/microsoft/sarathi-serve. Reports concrete gains: 2.6× higher serving capacity for Mistral-7B on one A100, 3.7× for Yi-34B on two A100s, and up to 5.6× on Falcon-180B.
AI-written summary · about
chapters (9)
- 0:00Sarathi-Serve intro
- 1:26Latency issues demo
- 3:09Latency-throughput tradeoff
- 7:25Mixed batching limits
- 9:09Compute-memory balance
- 10:56Stall-free chunked prefills
- 12:01VM vs Sarathi-Serve
- 13:48Quantitative evaluation
- 14:55Summary and adoption
related talks
16mRevisiting Pipeline Parallelism for LLM Serving
14mJITServe: SLO-aware LLM Serving with Imprecise Request Information
14mFastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
26mEfficiently serving LLMs at scale.
28mSponsored Session: Arctic Inference: Breaking the Speed-Cost Tradeoff in LLM Serving
15mResource Multiplexing in Tuning and Serving Large Language Models
14mToppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference
28mDatabricks' vLLM Optimization for Cost-Effective LLM Inference