Agentic Relevance Tuning With OpenSearch: Faster, Easier, and Higher Quality Search at Scale
summary
A new end-to-end system integrates OpenSearch User Behavior Insights, Search Relevance Workbench and an OpenSearch Agent Server to run a multi-agent pipeline that detects underperforming queries, proposes DSL-level hypotheses, and evaluates them automatically. The demo runs a 150-query offline eval (NDCG/MRR/etc.), finds a field-boosting hypothesis that improves metrics, and stops short of automated online A/B deploys (roadmap item). The code ships in OpenSearch 3.6, is open source, and invites contributors.
AI-written summary · about
chapters (10)
- 0:00Agentic relevance tuning intro
- 2:10Why relevance tuning matters
- 6:00Manual tuning challenges
- 8:07Closed-loop relevance improvement
- 11:41Introducing OpenSearch relevance agent
- 14:08What is OpenSearch Agent Server
- 17:47Multi-agent architecture
- 22:50Live demo: relevance agent workflow
- 33:25Key benefits and roadmap
- 37:27Q&A
related talks
39mIntroducing OpenSearch’s Search Relevance Workbench!
36mBuilding Next-gen Search With OpenSearch
32mPersonalisation and Specialisation of Search With OpenSearch Agentic Search
21mPersonalize Search Results with OpenSearch Agentic Memory
37mAgentic AI Meets Search: Real-World Use Cases With...
40mBuilding Intelligent Search Systems With Agentic RAG and OpenSearch
38mAI-Ready Data Foundation Using Amazon OpenSearch
22mAgentic Hybrid Retrieval: Orchestrating LLM Agents, Hybrid Search & Explainability in OpenSearch