Exploring the Mechanisms of LLM Reasoning Capabilities
summary
Presentation argues that LLMs often produce the appearance of reasoning but mainly rely on pattern matching and memorization, not robust logical inference. Empirical tests include counterfactual arithmetic and code tasks (base-10 vs base-9/16 and the thpy one-based language), the GSM-no-op variant of Great School Math showing accuracy drops of 17–65%, and MIT/Boston and Princeton analyses on premise order and sequence probability. Key failure modes are compounding multi-step errors, sensitivity to premise order and negation, dataset leakage tracked by the LM contamination index, while Chain of Thought prompting improves but does not solve compositional generalization.
AI-written summary · about
chapters (13)
- 0:00LLM reasoning intro
- 1:55Defining reasoning
- 3:25Logical reasoning types
- 5:56Why reasoning matters
- 7:47Deductive limits
- 10:18Deductive research
- 13:12Inductive challenges
- 15:27Abductive weaknesses
- 16:31Evaluating reasoning
- 19:02Input sequence impact
- 21:14Chain of Thought
- 22:21Reasoning limits summary
- 24:05Q&A
related talks
29mLLMs that think: Demystifying Reasoning Models
29mThe Evolution of LLMs: From Imitation to Reasoning
25mWhy LLMs Still Fail: Reasoning, Robustness, and Safety
01h 02mOut-of-Distribution Generalization as Reasoning: Are LLMs Competitive?
31mBeyond LLMs and Maturing RAG use cases
21mThe Coherence Trap: Why LLMs Feel Smart (But Aren’t Thinking)
28mSymbolic Reasoning in telecom oriented LLMs
01h 18mThe Enigma of LLMs: on Creativity, Compositionality, Pluralism, and Paradoxes