Interpretability For Everyone
summary
Shows that saliency maps fail sanity checks by barely changing even after networks are randomized, so visual explanations can be misleading; develops TCAV (Testing with Concept Activation Vectors) that represents a human concept as a direction in activation space and scores it via directional derivatives, producing a TCAV score as fraction of inputs (e.g., 0.8). Applies TCAV to diabetic retinopathy, finds models sometimes use unexpected concept HMA when accuracy is mediocre, prompting relabeling and practical debugging.
AI-written summary · about
chapters (16)
- 0:00Technical Vision Talks
- 4:02NON-goals
- 5:29Investigating post-training interpretability methods.
- 6:05One of the most popular interpretability methods for images: Saliency maps
- 6:49Sanity check question.
- 7:16Some confusing behaviors of saliency maps.
- 8:16Sanity check When prediction changes, do explanations change? No!
- 12:04Defining concept activation vector (CAV) Inputs
- 13:00TCAV core idea: Derivative with CAV to get prediction sensitivity
- 13:57Quantitative validation: Guarding against spurious CAV
- 14:51Results
- 15:12Collect human doctor's knowledge
- 15:32TCAV for Diabetic Retinopathy
- 16:19Responses from inside of academia
- 17:25Limitations of TCAV
- 18:55Things to keep in mind during our journey.
related talks
48mInterpretability - now what?
01h 00mData, visualization, and designing AI
50mExplainable AI Analysis & Visualization: Applications from Brain Computer Interfaces to ChatGPT
35mDeep Learning Model Interpretability for Computer Vision based Models
45mExplainable Machine Learning Explained
09mQuantitative Evaluation of Machine Learning Explanations: A Human-Grounded Benchmark
39mConceptual Challenges In Connecting Interpretability And Causality
37mInteractive Explainable AI