dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving
Talkat 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24) · 2024-07-10..07-12·
summary
dLoRA is a LoRA-serving system that dynamically merges and unmerges adapters and migrates both requests and adapters to reduce latency and boost GPU utilization. It implements credit-based batching and request-adapter co-migration plus cross-adapter batching to let different LoRA adapters share pre-trained weight computation. Built on 4 nodes (8 A800 GPUs each) serving 128 adapters for LLaMA-2 with ShareGPT traces, it reports up to 57.9× throughput vs vLLM, 26× vs HuggingFace PEFT and 1.8× lower latency than S-LoRA.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 0:38LoRA and its serving challenges
- 3:37Identifying serving inefficiencies
- 4:29The dLoRA system architecture
- 14:23Conclusion
related talks
32mLoRAX: Serve 1000s of Fine-Tuned LLMs on a Single GPU
14mToppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference
25mMulti Adapter Hosting LoRA Adapter
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...
29mLeveraging open-source LLMs for production
25mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
32mBeyond Round-Robin: GPU-Aware Load Balancing for LLM Inference in Kubernetes
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs