Parrot: Efficient Serving of LLM-based Applications with Semantic Variable
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
Parrot is an LLM-serving system that exposes Semantic Variable annotations so the service sees the dataflow across multiple prompt requests and can do end-to-end scheduling, batching, prefix sharing and engine placement instead of blind per-request optimization. A prototype implements DAG-based scheduling, shared-prefix detection and an optimized attention kernel; evaluations report up to an order-of-magnitude efficiency gain, 2.3× speedup for meeting recap, >2× for map-reduce workflows, up to 1.7× latency and 12× compute improvements in some kernels, and 2.4× throughput by avoiding redundant work.
AI-written summary · about
chapters (5)
- 0:00Modern LLM application workflows
- 2:25Limitations of current services
- 5:51Introducing Parrot and abstractions
- 7:54Optimizing LLM serving
- 10:48Evaluation and conclusion
related talks
14mFastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
14mJITServe: SLO-aware LLM Serving with Imprecise Request Information
20mSGLang: An Efficient Open-Source Framework for Large-Scale LLM Serving
28mSponsored Session: Arctic Inference: Breaking the Speed-Cost Tradeoff in LLM Serving
26mMastering Large Language Models: Prompt Engineering for Precision & Performance
24mverl: An Open Source Large Scale LLM RL Framework for Agentic Tasks
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
33mParallel Processors: Past & Future Connections Between LLMs and OS Kernels.