Weaver: Efficient Multi-LLM Serving with Attention Offloading
summary
Weaver implements attention offloading that lets hot models borrow KV cache memory on cold-model GPUs by shipping QKV vectors and running attention there, avoiding full model duplication. It implements a GPU-driven control flow (lightweight polling kernels and shared memory counters) and operator splitting (greedy kernel halving) to avoid head‑of‑line blocking. On A100 and L40S hardware with real LLM traces it boosts hot-model throughput up to 77% while adding only 3–5 ms TPOT to cold models.
AI-written summary · about
related talks
15mvLLM: Easy, Fast, and Cheap LLM Serving for Everyone
26mEfficiently serving LLMs at scale.
28mPagedAttention: Revolutionizing LLM Inference with Efficient Memory Management
26mLMCache: Lower LLM Performance Costs in the Enterprise
16mWLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
14mFastServe: Iteration-Level Preemptive Scheduling for Large Language Model Inference
34mBeyond Static Pods: Dynamic GPU Sharing and Low-Latency Model Switching for LLM ...
11mLightning Talk: Inside VLLM's KV Offloading Connector: Async Memory Transfers for...