Lightning Talk: The Perception Test
Lightningat Understanding Higher-Level Intelligence from AI, Psychology, and Neuroscience Perspectives · 2024-06-24..06-28·
summary
DeepMind builds a new benchmark and films 11,600 videos with 100+ crowdworkers, rich annotations (VQA, object and point tracks, action and sound segments) to probe object permanence, physics, memory and multimodal errors rather than just semantics. Off‑the‑shelf models (Flamingo, SeViLA, then GPT‑4V and Gemini) score about 55% vs >90% human on multiple‑choice VQA, and models perform worse than random on counterfactual questions, showing persistent gaps despite progress. Data and challenge are public.
AI-written summary · about
chapters (13)
- 0:00Perception Test project
- 1:51Limitations of benchmarks
- 3:40Designing real-world benchmark
- 7:13Filming diverse video data
- 9:03Object permanence with cups game
- 11:54Stability and adversarial tests
- 13:20Multi-modality with audio-visual tests
- 14:25Model biases with adversarial examples
- 15:50Benchmark structure and metrics
- 16:52Results and human gap
- 18:19Data contamination challenges
- 19:46Closing remarks
- 20:06Q&A
related talks
01h 03mIt's All About AI
01h 05mCan you trust your (large language) model?
29mLies, damned lies and large language models
17mPsychology of AI
45mPanel Talk
58mMirror, mirror: LLMs and the illusion of humanity
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
16mNavigating the Uncharted: Ensuring Prompt Quality in the Age of Language Models