Discussion
Talkat Understanding Higher-Level Intelligence from AI, Psychology, and Neuroscience Perspectives · 2024-06-24..06-28·
summary
Panelists argue that lots of capabilities emerge from models trained on code+text, with concrete examples: variable binding and long-range reference appearing only after code exposure, and syntax and coding skills surfacing at far less data than expected — roughly 100 million vs 500 billion tokens for basic grammatical competence versus richer abstractions. Theory claims (a sequential density estimator view) explain how rollouts can sew together local expertise into reasoning. Missing pieces include episodic memory; ad hoc memory stores and multi-agent demos (a simulated town / Valentine’s Day) show practical fixes that already work.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:05Theory in cross-modal learning
- 7:35Language for reasoning
- 15:18Wittgenstein on language
- 22:57Language-only training results
- 31:07Next-word prediction vs intent
- 34:26Optimization tricks and future models
- 38:40Backwards training and memory
related talks
01h 07mDiscussion Session
33mSkill Issue
10mWhat goes into the making of LLMs
01h 05mAdaptive Generalization: The Role of Dynamic Evaluation in Low-Resource Settings
30mOpen Challenges for Programming Agents
46mEvaluating Patterns in Gen AI Outputs
01h 11mThe Move Toward AGI: Why Large Language Models Surprised Almost Everyone... | Theoretically Speaking
34mProgramming Languages for AI Programing Agents