Social Behavior Prediction from Video Observations
Talkat Understanding Higher-Level Intelligence from AI, Psychology, and Neuroscience Perspectives · 2024-06-24..06-28·
summary
Proposes predicting 3D human behavior by regressing off-the-shelf 3D parametric body models and using a discrete motion vocabulary learned with a VQ‑VAE motion codebook so transformers can operate on tokens. Introduces a poly‑autoregressive prediction scheme to condition one agent on many others, trains on hours of face‑facing interactions scraped from talk shows and Zoom talk dataset sessions, synthesizes listener videos, and reports human evaluators prefer the model over strong baselines and often cannot tell model from real.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 1:48Defining social behavior prediction
- 4:17Visual prediction tokens and 3D representations
- 6:503D human behavior prediction
- 7:32Nonverbal communication in interaction
- 9:00Text-only AI limitations
- 11:27Multimodal prediction challenges
- 12:33Unsupervised learning revolution
- 13:17Autoregressive prediction models
- 15:08Swing dance demo
- 15:51Listener motion from speaker cues
- 17:14Zoom data for nonverbal communication
- 19:19Synthesizing listener responses
- 20:00Motion codebook representation
- 22:34Transformer prediction with motion tokens
- 23:40Evaluating motion realism
- 24:22Infant communication modeling
- 25:24Text-to-motion with language models
- 27:56Multi-agent prediction
- 29:02Brain mechanisms and animal behavior
- 29:48Summary and future directions
related talks
18mControllable Generation of Synthetic Video Data With Human Motion
06mMultimodal Joint Head Orientation Estimation in Interacting Groups via Proxemics and Interaction ...
03mBuilding artificial people: Endless possibilities and the dark side
20mPopup: reconstructing 3D video using particle filtering to aggregate crowd responses
23mAVA: Towards Agentic Video Analytics with Vision Language Models
35mDeep learning for 3d human motion capture
41mComputer Vision for Remote AR
27mAnalysis of sports scenes using computer vision models