How grounded are ungrounded LLM models
Talkat Understanding Higher-Level Intelligence from AI, Psychology, and Neuroscience Perspectives · 2024-06-24..06-28·
summary
Finds that strong text-only embeddings often remove the added benefit of pixels on parsing benchmarks, shows a pipeline where an LLM generates scene graph outputs that guide diffusion models to produce correct attribute binding (gray cat/orange dog, man in red/woman in blue) using only dozens of GPUs, demonstrates multimodal tools (LLAVA, SAM) that segment and reason, presents a vision-only Llama-style model trained on pixel tokens, and introduces a control model (humanoid locomotion as next-token prediction) that runs and generalizes in simulation and real environments.
AI-written summary · about
chapters (9)
- 0:00Grounding llms in perception
- 4:28Multimodal parsing
- 9:20Scene layouts and graphics
- 16:09Human pose priors
- 20:41Segmentation models
- 24:11Vision and control models
- 30:48Robotic control trajectories
- 34:11Larva robotic framework
- 39:06Discussion and conclusion
related talks
01h 19mBuilding an LLM on your Laptop to...
27mThe Cutting Edge of AI: From LLMs to Multimodal Intelligence
35mReading the Mind of an LLM
52mGeneralization in the representations and computations of frontier language models.
07mLightning Talk: Grounding LLM to Avoid Hallucinations Using Vector S...
32mEverything you wanted to know about VLMs but were afraid to ask
28mThe Data That Shapes Foundational LLMs
46mUnder the Hood of LLMs — all those questions you never dared ask about them