Beyond Scoring: Rethinking Ranking in the LLM Era
summary
Modern ranking swaps handcrafted objective weights for natural language objectives and shows how to combine reasoning with behavioral prediction: start with prompting a base model, teach platform-specific signals via supervised fine-tuning, refine with pairwise preference data using direct preference optimization (or RLHF), then compress with knowledge distillation and serve cheaply via prefix caching and cascading/routing. The talk emphasizes concrete trade-offs — latency, validation, multi-objective constraints (hard vs soft) — and evaluates reasoning with LLM judges plus traditional NDCG-style metrics.
AI-written summary · about
chapters (25)
- 0:00Setting the Stage: What Changes When LLMs Enter Ranking Systems
- 1:07Speaker Intro - Rhea Goel, Applied Science Manager, Amazon
- 2:08Three People, One Hotel Query, Three Correct Rankings
- 3:10Ranking Has Never Been About Finding One Best Item
- 3:41Today's Architecture: Retrieval, First-Stage Ranker, Re-Ranker
- 4:12When Product Requirements Become Ambiguous English Sentences
- 4:42What If We Optimized for Language Instead of Objective Functions?
- 5:43Five Questions Before Replacing a Re-Ranker With an LLM
- 6:13Starting Simple: Prompting a Base Model
- 7:15Impressive Output, but Would You Deploy It Tomorrow?
- 7:45Behavioural Models Predict Actions; LLMs Reason About Intent
- 8:46Supervised Fine-Tuning: Teaching the Model Your Platform
- 9:47When Imitating History Is the Wrong Objective
- 10:48Preference Data: Asking Which Ranking Is Better
- 11:50RLHF, Reward Models and the Learned Objective Function
- 12:22Direct Preference Optimization as the Simpler Path
- 13:57Multi-Objective Reality: Affordability, New Inventory, Revenue
- 14:29Hard Constraints vs Soft Constraints and Consistent Reasoning
- 16:32Making It Fast Enough: Knowledge Distillation to a Student Model
- 18:07Prefix Caching, Routing and Cascading
- 20:10Evaluating Reasoning: Beyond NDCG and MRR
- 20:40LLM as a Judge With Rubric, Chain of Thought and Structured Output
- 22:48Four Engineering Tensions That Frame the Whole Design
- 24:20The Real Question Isn't Replacement - It's Combination
- 25:23Q&A: Agentic Re-Rankers, Two Correct Answers, Continuous Improvement
related talks
29mBuilding Learning to Rank models for search using LLMs
30mLLM-Based Recommendation Systems
31mBeyond LLMs and Maturing RAG use cases
33mFrom LLM-as-a-Judge To Human-in-the-Loop: Rethinking Evaluat...
29mLLMs that think: Demystifying Reasoning Models
28mNavigating Challenges and Enhancing Performance of LLM based Applications
32mA Practical Guide to LLM Evaluation
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications