LoRAX: Serve 1000s of Fine-Tuned LLMs on a Single GPU
summary
LoRAX claims to pack 1000s of LoRA adapters on a single GPU by keeping a base model resident and swapping tiny LoRA checkpoints just-in-time, with first‑time loads around 100 ms and GPU resident adapters batched for inference. It implements dynamic adapter loading, an adapter exchange scheduler, and heterogeneous continuous batching that keeps throughput nearly constant even with up to 128 adapters per batch, cutting cost so a single A10G handles dozens of fine‑tuned models versus one‑model‑per‑GPU deployments.
AI-written summary · about
chapters (19)
- 0:00LoRAX introduction
- 1:04Fine-tuned vs general LLMs
- 3:53Serving multiple models
- 6:21Reasons to fine-tune more than one model
- 7:25Cost challenges
- 8:30Parameter efficient fine-tuning
- 10:56Efficient adapter serving
- 13:20Dynamic adapter loading
- 15:46Heterogeneous batching
- 18:36Performance gains with CUDA
- 20:45Additional LoRAX optimizations
- 22:11OpenAI compatibility
- 22:51Structured output generation
- 24:15Speculative decoding
- 25:40Mixture of adapters
- 27:26Production systems
- 28:50Closing remarks
- 29:13Q&A: CPU vs GPU
- 30:32Q&A: Beyond text models
related talks
15mdLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving
14mToppings: CPU-Assisted, Rank-Aware Adapter Serving for LLM Inference
25mMulti Adapter Hosting LoRA Adapter
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs
15mResource Multiplexing in Tuning and Serving Large Language Models
22mTraining a LLaMA in your Backyard: Fine-tuning Very Large...
32mBeyond Round-Robin: GPU-Aware Load Balancing for LLM Inference in Kubernetes
22mPerformance Optimizations at 100K+ Scale