On cognitive maps, LLMs, world models, and understanding
summary
Claims that space is a latent sequence and that hippocampal “place cells” reflect sequence-conditioned latent states rather than GPS-like locations; introduces clone-structured causal graphs (an overcomplete HMM with deterministic emissions/clones, POMDP actions and a vector quantizer) that learns graphs from aliased visual streams, reproduces rodent latent‑learning dynamics better than LSTMs/transformers, produces place‑cell‑like units, supports planning, rapid surprise‑driven rebind on surprise adaptation, schema transfer across rooms, and simple algorithmic in‑context rebinding — and the model empirically works on navigation and transfer tasks.
AI-written summary · about
chapters (13)
- 0:00Introduction to speaker
- 1:02Hippocampus and latent learning
- 3:24Rethinking place cells
- 6:12Clone-structured causal graphs
- 9:11Sequence learning in space
- 12:31CSCG model details
- 20:41Advantages of latent structures
- 23:43Schema learning and transfer
- 27:21Rebinding and planning
- 33:02In-context learning mechanisms
- 39:05Transformers and latent models
- 47:41Philosophy of understanding
- 59:52Q&A and closing remarks
related talks
52mGeneralization in the representations and computations of frontier language models.
31mIs My Brain a Knowledge Graph? A Human Take on Connecting the Dots
35mReading the Mind of an LLM
41mEnhancing LLMs with Graph Technology
49mHow grounded are ungrounded LLM models
44mAI Intuition: Exploring Language Model Latent Space
34mCausality, robustness, and natural language understanding in ML
54mTemporal Context in Brains and AI