Untangling Real Products and Hallucinations, Postmortem of a Phone Agent Sock Finder
summary
A phone-agent for finding items in a 20,000‑product medical hosiery catalog faced ASR errors on technical terms and LLM hallucinations once candidate lists passed about 70 items. The solution replaces a single prompt+vector-search approach with Algolia fuzzy search plus a verifier LLM and a three‑agent pipeline: a product‑reconstruction LLM, retrieval via Algolia, a conversational LLM to narrow results to 10–20 candidates, and a verifier LLM, reducing hallucinations and latency enough for deployment while balancing cost and context‑window limits.
AI-written summary · about
chapters (8)
- 0:00Sock history intro
- 2:34ML limits in NLP
- 3:35Phone product search challenge
- 7:10LLM constraints and hallucinations
- 9:16Vector DB failure and RAG
- 10:20Algolia and agent solution
- 12:23Lessons learned
- 13:47Post-deployment improvements
related talks
21mUntangling Real Products and Hallucinations: Postmortem of a Sock-Finding Call Agent
21marchive – Clara Zang
28mRetrieval-Augmented AI Agents for Entity Matching at Scale
32mBuilding Multi-Agent RAG Systems
26mReducing Hallucinations and Evaluating LLMs for Production
17mVectors Vs. Hallucinations: OpenSearch's GenAI Survival Kit
10mAnalyzing 10,000 Sales Calls With AI In 2 Weeks
22mAgentic Hybrid Retrieval: Orchestrating LLM Agents, Hybrid Search & Explainability in OpenSearch