Knowledge is embedded in language neural networks but can they reason?
summary
Large transformer language models embed surprising world knowledge but fall short on multi-step reasoning. Scaling (e.g. 117M→1.5B parameters, training on tens of billions of tokens) yields GPT‑2 continuations that place Cincinnati with the DOE and BERT/BERT‑like pretraining drives SQuAD to ≈ human performance, yet copying from training is low (~3%) and models still make factual reasoning errors (BERT answers “Ming” instead of “Song”). Prototype fixes include graph‑based neural state machines and MAC modules that build scene graphs and iterate to answer VQA queries (e.g. find the red apple).
AI-written summary · about
chapters (16)
- 0:00Intro
- 2:47Tree-structured models
- 4:33Language Modeling
- 6:05LMs in The Dark Ages: n-gram models
- 8:00Enlightenment era neural language models (NLMs)
- 11:09GPT-2 language model (cherry-picked) output
- 18:58Transformer models
- 21:32Classic Word Vec (Mikolov et al. 2013)
- 25:54Self-attention in (masked) sequence model
- 37:01Good systems are great, but still basic NLU errors
- 40:46What is Reasoning? Bottou 2011
- 42:10Appropriate structural priors
- 42:56Compositional reasoning tree
- 43:22A 2020s Research Direction
- 44:07A Neural State Machine
- 46:48NSM accuracy on GQA
related talks
27mLarge Language Models and the Future of AI
29mHow AI Models Get Smarter
43mBeyond the Hype: A Realistic Look at Large Language Models
30mLarge Language Models and Prompt Engineering for Developers
41mRepresentations from natural language data: successes and challenges
01h 00mInside GPT – Large Language Models Demystified
39mUnlocking reasoning in Large language models
01h 05mCan you trust your (large language) model?