Designing Fast & Cost-Efficient Agents: The Power of Caching
summary
Claims that straightforward caching cuts LLM agent cost and latency while preserving capability and shows concrete results: a cited $6.80 per complex agent task estimate, a demo saving ~180 tokens and cutting latency to ~25% of an LLM call, and Redis customers seeing >50% cache-hit averages. Explains prompt caching (KV/attention cache to avoid recomputing context), semantic caching (embeddings + cosine thresholding, cross‑encoder re-rank and sampled LLM judge), and a built product LangCache with a tiny tuned embedding model.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:19LLM inference bottleneck
- 3:50Token consumption in loops
- 4:54Caching core principle
- 9:55Prompt caching benefits
- 10:59Semantic caching
- 23:16Live demo: semantic caching
- 25:49Cache management & privacy
- 27:57Security & user data
- 29:18Cache invalidation
- 30:02Q&A
related talks
14mOptimizing RAG with Semantic Caching & LLM Memory
34mVLLM KV Cache Management: From Cache Reuse To Agent Scenario Optimization
24mCost and Performance Optimization of LLM Inferencing.
36mOptimizing LLM Performance With Caching Strategies in OpenSearch
17mCortex: Achieving Low-Latency, Cost-Efficient Remote Data Access For LLM via
16mAgentix: An Efficient Serving Engine for LLM Agents as General Programs
26mLMCache: Lower LLM Performance Costs in the Enterprise
14mMake Agents Remember the Right Context at the Right Time