Lightning Talk: Not All Tokens Are Equal: Semantic KV-Cache for Agen...
summary
A runtime called Semantic KV-Cache replaces anonymous KV blocks in llm-d and vLLM with typed state that tags tokens as SystemPrompt, ToolDefinition, ReasoningBranch and applies differentiated policies: pinning foundational context, replicating shared tools, eagerly evicting ephemeral reasoning. It demonstrates lifecycle-aware retention and movement across GPUs, integrates ideas from Continuum and KB-Flow for multi-agent pipelines and large 30k–100k shared prefixes, and shows reduced recomputation and lower “Agentic Tax” in their experiments.
AI-written summary · about
chapters (5)
- 0:00KV-centric inference intro
- 1:12Request-centric inference today
- 3:42State blindness in agents
- 6:07Semantic KV-cache policies
- 9:46Key takeaways and future
related talks
34mVLLM KV Cache Management: From Cache Reuse To Agent Scenario Optimization
27mShattering the AI Memory Wall
25mTo Cache or Not To Cache? a Tiered KVCache Storage System for Agent ...
11mLightning Talk: Intelligent Traffic Routing for Distributed LLM Inference: Beyond Trad...
15mKV Cache: Accelerating AI Inference on Intel CPU
19mBeyond the Node: Scaling Inference with Cluster-Wide KVCache...
26mAmazingly Fast and Incredibly Scalable Inference...
52mReduce LLM calls with Vector Search Design Patterns