The Brain’s Guide to Dealing with Context in Language Understanding
summary
Argues that language regions form a hierarchy of timescales and shows that modern models mirror this: compares LSTMs, temporal convolutional network (TCN) and attention models, measures unit activations against fMRI voxels using ridge regression, finds TCNs and transformer XL produce layer-specific mappings to brain regions and improve perplexity on WikiText‑103, shows LSTMs fail to capture multi-timescale hierarchy, and highlights mysterious drop in predictivity after layer ~9 despite strong mid‑level alignment.
AI-written summary · about
chapters (20)
- 0:00Intro
- 1:03Natural Language Understanding
- 1:37A 1960's example
- 3:13Applications of NLU
- 4:29The importance of context in language understanding
- 5:14Key questions for this talk
- 6:07The brain's organization
- 10:36Evaluating the performance of these models
- 12:23Example sequence modeling tasks
- 14:18Using recurrence to solve the problem
- 16:30Dealing with longer timescales
- 23:23Temporal Convolutional Neural Networks
- 27:01Transformer Networks
- 35:31The methodology
- 36:32Predicting brain activity with encoding models
- 36:45Relative predictive power of models
- 38:04Open questions
- 38:24Layer-specific correlations for Transformer
- 39:14Encoding model performance for Transformer
- 40:13Summary and Challenges
related talks
01h 30mToward Understanding In-context Learning
01h 26mFrom RNNs to Transformers: A Journey through Language Modeling
46mContext is all you need: FUNdamental linguistics for NLP
25mLooking beyond LSTMs: Alternatives to Time Series Modelling using Neural Nets
45mMachine Learning, Deep Networks and Interpretability in Systems, Cognitive and...
47mInterpreting Deep Neural Networks (DNNs)
01h 01mStructural Transfer Learning: Exploring Neural Models and Language Structure Through...
43mBeyond the Hype: A Realistic Look at Large Language Models