Toppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference
summary
System pins the base LLM on GPUs and dynamically loads LoRA adapters from host memory, while CPU-assisted LoRA computation precomputes the lightweight XAB adaption on CPUs during adapter transfer, using shared-memory zero-copy and multiple isolated CPU processes to avoid sync and threading overheads; a pipeline switches to GPU to finish decoding. A linear performance model drives rank-aware scheduling across servers. Evaluation on 100 GPUs shows up to 1.7× lower latency and SLO attainment up to 99%; scaling to 512 adapters exposes high loading cost that this design mitigates.
AI-written summary · about
chapters (11)
- 0:00Toppings system intro
- 1:01LLM serving workflow
- 2:45Multi-tenant LoRA challenges
- 4:32Adapter loading overhead
- 5:58Heterogeneous LoRA ranks impact
- 7:25Toppings architecture
- 8:07CPU-assisted LoRA design
- 9:54Rank-aware scheduling
- 11:00Evaluation setup
- 12:22Production workload results
- 13:30Summary
related talks
32mLoRAX: Serve 1000s of Fine-Tuned LLMs on a Single GPU
15mdLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving
25mMulti Adapter Hosting LoRA Adapter
32mBeyond Round-Robin: GPU-Aware Load Balancing for LLM Inference in Kubernetes
14mScalable Solutions for Running Large Language Models
34mHow To Evolve Your LLM Self-Hosting Platform: A Practical Guide To Adopting…
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone