How to Fail Interpretability Research
summary
Field surveys identify 11 recurring ways research stalls and offer concrete checks and datasets to avoid them, for example a synthetic attribution test that pastes a pasted Corgi dog patch into scene images so any method highlighting the dog is a false positive, and a saliency sanity check where randomizing weights leaves similar maps. Papers often lack rigor: 40% qualitative-only results, 60% no human experiments, 17% invoke trust without evidence, 40% skip a clear definition.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:45Premeditation of evils
- 4:04Interpretability hype.
- 4:32well... we've been here before...
- 5:15What this talk is about
- 6:35What this talk is NOT about
- 8:19We need interpretability to increase user trust.
- 12:24We need to understand every single bit of the model.
- 15:51Agenda Many opportunities to fail.
- 18:33The performance and interpretability trade-off is inevitable.
- 21:39How I present the explanations doesn't
- 24:03Since there is no good way to evaluate interpretability methods, I can only show you qualitative results
- 32:16I am a computer scientist! Running human
- 33:25The explanation is always true; It is what the model thinks.
- 39:09I'm just a researcher who provide technical tools. The real world usage is something I cannot control.
related talks
42mQuestions and Discussion
41mWhere Have All the Metrics Gone
23mHow to fail AI implementations
29mFive things I learned from turning research papers into industry prototypes
53mPlenary Discussion: How Can One Test/Falsify Current Theories of Brain Computation?
30mReproducible Machine Learning
01h 04mCan you trust your (large language) model?
41mIf We Want AI to be Interpretable, We Need to Measure Interpretability