LLM skills and meta-cognition: scaffolding for new forms of learning?
summary
Shows that frontier models actually extract and name procedural skills and can compose them: GPT-4o reliably generates novel skill labels and composes up to five randomly chosen skills in synthetic tests, and Gemini 2.5 Pro produces surprising embodied skill names (e.g., “edge-aware placement”). Using those labels to synthesize 4,000 SFT QA pairs raises a Llama3 8B base to Alpaca-like performance. A phrasebook curriculum with 20% context dropout lets transformers learn deep multi-step translations that plain IO training fails to acquire.
AI-written summary · about
chapters (18)
- 0:00LLM skills and metacognition intro
- 1:09Next-word prediction limits
- 2:14Skills as learned procedures
- 4:02Complex reasoning procedures
- 5:57LLM metacognition exploration
- 7:04Eliciting metacognition
- 9:57Skill extraction method two
- 13:56Next-word loss and metacognition
- 18:18Synthetic instruction data
- 20:35Better instruction tuning
- 23:04Skill composition framework
- 29:55Pretraining and skill tests
- 33:15Stronger skill composition
- 41:01Context learning and chain-of-thought
- 47:21How context enhances learning
- 55:28Metacognition research questions
- 58:41AI safety composition risks
- 1:00:47Theory vs real-world complexity
related talks
01h 04mLLM Metacognition: Understanding and leveraging "Thinking about Thinking"
01h 18mThe Enigma of LLMs: on Creativity, Compositionality, Pluralism, and Paradoxes
19mLLMs reasoning and agentic capabilities over time
37mExpanding the Capabilities of LLMs Through Community Collaboration
29mBeyond Solo: Enhancing LLMs through Collaborative Dialogue with GAN-Inspired Approaches
44mUnderstanding the abilities of AI systems: Memorization, generalization, and points in between
01h 39mMastering LLMs: From Prompt Engineering to Agentic AI
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence