Synergy of Signals: Traffic Logs Meet LLM Labels
summary
E‑commerce search relevance is built by merging user traffic logs with LLM labels to improve learning‑to‑rank targets: unknown traffic scores get LLM checks, LLM‑irrelevant can set score 0 or low (1) for sampled items, and for traffic‑relevant items a purchase threshold makes the pipeline trust logs while below that it defers to the LLM to correct misclicks. The system labels under <10% of terms via stratified sampling, yields 70% unchanged 30% increased sessions with more relevant products, uses English priming plus Romanian examples, and validates changes with live A/B tests showing positive results.
AI-written summary · about
chapters (8)
- 0:00Defining relevance
- 1:07Data sources for relevance
- 2:30Role of LLMs in labeling
- 4:35Comparing traffic and LLMs
- 7:38Unified labeling process
- 8:58Combining label sources
- 10:20Model performance and future
- 13:14Q&A and best practices
related talks
29mBuilding Learning to Rank models for search using LLMs
20mBeyond Relevance: Using LLMs to Transform E-commerce Search into Product Discovery Engines
01h 30mUsing LLMs to improve your Search Engine
25mTransforming E-Commerce with LLMs: 3 Real-World Use Cases
21mTransforming search and discovery using LLMs
36mOptimizing LLM Performance With Caching Strategies in OpenSearch
30mLLM-Based Recommendation Systems
24mConversational Search with LLMs