The Frontier between Retrieval-augmented and Long-context Language Models
summary
Presentation argues that retrieval-augmented models still lack a reliable training recipe because joint updating of encoder and datastore forces costly reindexing, while long-context models advance faster; claims include Retro’s old result (comparable to GPT‑3 with 25x fewer parameters) and a stalled research community on RAG. Results show an 8B ProLong model trained with ~40B tokens to 512k context works well if continued pretraining mixes long and short data. New evaluations include HELMET for downstream long-context tasks and ELK for citation quality.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:51Retrieval vs long-context definitions
- 3:23Why retrieval matters
- 8:00Efficiency limits of RAG
- 9:26Attribution challenges in RAG
- 12:35Need for long-context models
- 15:24Can long-context replace RAG?
- 17:53Training challenges in RAG
- 21:47Why RAG progress stalled
- 30:17Evaluating long-context models
- 33:53HELMET benchmark
- 39:37What fails in long-context training
- 41:03Training recipe for long context
- 42:27ProLong approach
- 46:39ProLong key findings
- 51:44Context length impact
- 53:34Instruction tuning for long context
- 53:55Long-output tasks focus
- 57:10Summary
related talks
51mLong context modeling and generalization: two perspectives
49mContext is King
30mLarge Language Models and Prompt Engineering for Developers
01h 30mToward Understanding In-context Learning
30mRAG: Tricks from the trenches.
15mMemory-Augmented AI Agents: Enhancing Long-Term Reasoning and Context Awareness
02h 50mUnderstanding and Using AI Language Models
19mRAG in 2025: State of the Art and the Road Forward