Project Aristo: Towards Machines that Understand Science
summary
Project Aristo is a system that answers Grade 8 New York Regents non‑diagram multiple‑choice science questions, reaching over 90% after adding neural language models where earlier systems scored under 60% and classical versions hit ~36–63. It ensembles retrieval of 20–30 science sentences with pretrained language models, shows strong handling of negation and conjunction (94% and up to ~98% on probes) but counts only 6% on counting tasks. Failures mainly stem from missing retrieval, subtle comparison errors (light years vs astronomical units), and inability to model world state or do systematic multi‑step decomposition; RuleTaker-style training yields principled counterfactual reasoning.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:26Science Questions: A Grand Challenge...
- 4:05Progression on NY Regents 8th Grade (NDMC)
- 5:50Aristo: an over-simplified overview
- 6:26Language Models
- 11:553. More than Pattern Matching?
- 13:41Synthetic Conjunction Test
- 16:214. Where is Aristo failing? . Case study on 30 failures
- 17:28Good support for the correct answer (13%)
- 18:34Good support for the incorrect answer (3%)
- 19:58Math Reasoning
- 21:081. Generative Models
- 23:222. Question Decomposition
- 24:30Modeling World States
- 27:01Systematic Reasoning and Instruction
- 30:52Summary and Reflections
related talks
17mNatural Language Processing in Automation
39mUnlocking reasoning in Large language models
19mAI: The current state of play
44mUnderstanding the abilities of AI systems: Memorization, generalization, and points in between
17mPsychology of AI
31mMachine Reasoning and System 2 Thinking
19mComplementing Political/Social Research with AI for a Deeper Understanding of Society
42mReasonable AI