Optimizing LLM Performance With Caching Strategies in OpenSearch
summary
Shows an OpenSearch-backed LLM cache that combines exact key and semantic matching, integrates with LongChain GPT-cache patterns and OpenSearch vector stores, and exposes a tunable scoring threshold (0–1). Real examples: an e‑commerce demo with $750/day LLM spend and a healthcare deployment that cuts LLM invocations by 50% fewer LLM invocations. Implementation details include 5s → 29ms cached latency, HNSW/IVF choices with HNSW + phase, JVM/vector memory and quantization, and eviction and warm‑up strategies.
AI-written summary · about
chapters (7)
- 0:00LLM caching intro
- 4:35Why caching matters
- 10:41Exact vs semantic caching
- 16:29OpenSearch memory caching
- 24:56Vector search optimization
- 32:19Cost benchmarking strategies
- 33:52Q&A
related talks
49mLLM—Search Engine Killer?
24mConversational Search with LLMs
17mCortex: Achieving Low-Latency, Cost-Efficient Remote Data Access For LLM via
46mLLM—Search Engine Killer?
22mAgentic Hybrid Retrieval: Orchestrating LLM Agents, Hybrid Search & Explainability in OpenSearch
30mDesigning Fast & Cost-Efficient Agents: The Power of Caching
01h 30mUsing LLMs to improve your Search Engine
14mOptimizing RAG with Semantic Caching & LLM Memory