Inherent Trade-offs with the Local Explanations Paradigm
summary
Compares gradient, SmoothGrad, Integrated Gradients, LIME and influence‑based explanations using concrete checks: reinitializing Inception‑v3 layers (model randomization test) and shuffling labels (data randomization test), then measuring rank correlations and AUC of feature‑removal curves. Finds some methods — notably guided backprop shows no change — remain visually and quantitatively similar after randomization, and adversarial perturbations can flip explanations without changing predictions. Shows local explanations leak models (influence rankings enable model extraction) and that DP‑SGD (per‑example gradient clipping + noise) forces a privacy vs. usefulness tradeoff as smaller epsilon makes explanations noisy.
AI-written summary · about
chapters (22)
- 0:00Intro
- 2:23Interpretability" AND "Neural Network
- 3:00Some Motivation
- 4:40Goals: Model Debugging
- 6:38Systematic Subgroup Errors
- 7:26Promise of Explanations
- 8:40'Interpretability' vs 'Explainability
- 13:28Saliency/Attribution Maps
- 17:23"Local Explanations" Local Explanations: focus on the behavior of the model around a single point.
- 18:55Key Takeaways Difficult to assess quality and model fidelity of local explanations
- 22:45Gradient/Sensitivity Map
- 23:13Smoothgrad
- 23:49Integrated Gradients
- 27:02Non-Image Settings
- 28:44Challenge 1: Assessment
- 30:42Sanity Checks for Saliency Maps
- 34:26Medical Setting
- 35:24Analysis & Visual Assessment
- 36:41Sanity Checks Useful?
- 37:05Attacks
- 42:47Model Recovery
- 43:14Privacy 'Harms': Examples & Prototypes
related talks
01h 05mEvaluating Model Explainability Methods
16mBuilding Trust through Explainable AI
14mThe effects of example-based explanations in a machine learning interface
01h 29mNavigating ML Explainability: From Theory to Implementation
17mExplaining models: an empirical study of how explanations impact fairness judgment
17mChallenges Towards Production-Ready Explainable Machine Learning
24mWhy and How to Build Explainability into your ML Workflow
28mManaging AI Risk: Beyond Explainability