Efficient Retrieval of Influential LLM Training Examples
summary
Argues that training-data attribution splits into an Estimation Problem (accurately estimating average causal effect) and a Retrieval Problem (finding top examples efficiently). Uses EKFAC inverse-Hessian estimators to push correlation on linear-data-modeling benchmarks from ~0.2–0.4 up toward ~0.6, and builds a generic sketching retriever (P‑SIFT) that projects preconditioned gradients (projection dims ~3M), cutting per-query work from weeks to hours and scaling to frontier models (used in Claude Mythos Preview safety tests).
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:38Motivation for training data counterfactuals
- 7:12Causal modeling of training interventions
- 13:42Influence functions estimation
- 27:00Retrieval problem and brute force limits
- 31:21Sketching for efficient retrieval
- 41:43Model safety applications
- 45:41Summary and open research
- 46:43Q&A
related talks
40mThe causality trap: why your LLM sounds smart but thinks shallow
01h 27mTutorial: Incentives for Collaborative Learning and Data Sharing, Part II
37mActive Teaching, Human Learning
19mHow to look at your data
37mLinear representations of concepts in modern AI models
35mReading the Mind of an LLM
34mAttention is not all you need - Keynote
56mWhat can be passively learned about causality?