VLLM KV Cache Management: From Cache Reuse To Agent Scenario Optimization
Talkat KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026 · 2026-09-07..09-09·
summary
vLLM presents techniques to reduce VRAM pressure from large KV caches in multi-turn LLM agent workloads by reusing and offloading cache entries. It describes Prefix Caching (block-hash reuse of prompts, RAG, and history) and a three-tier KV Offloading system (LRU eviction, CPU offload, and ARC policy) and discusses future ideas like attention-score-based KV compression, priority scheduling for agent dependencies, and cross-session prefix sharing.
AI-written summary · about
related talks
26mLMCache: Lower LLM Performance Costs in the Enterprise
17mCacheSlide: Unlocking Cross Position-Aware KV Cache Reuse for Accelerating LLM Serving
23mFast Inference, Furious Scaling: Leveraging VLLM With KServe
30mDesigning Fast & Cost-Efficient Agents: The Power of Caching
10mLightning Talk: KV-Cache Centric Inference: Building a State-Aware...
28mPagedAttention: Revolutionizing LLM Inference with Efficient Memory Management
11mLightning Talk: Inside VLLM's KV Offloading Connector: Async Memory Transfers for...
44mVLLM K/V Caching With Ceph