Shape n’ Swarm: Hands-on, Shape-aware Generative Authoring with Swarm UI and LLMs
summary
Proof-of-concept Shape n’ Swarm combines hand-arranged tabletop robots and speech to produce physical animations and interactions: a shape-aware generative authoring method using a multi-agent LM architecture that calls two custom APIs to map shapes (XY grid positions and automated groupings) to actuator targets. Animations appear within ~5 seconds; a workshop (n = 11) shows participants create diverse, code-free behaviors and task solutions. Failures stem from LLM misalignment and limited discrete kinematic control.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 1:10Shape-changing interfaces background
- 2:58Shape-aware generative authoring concept
- 3:19Authoring method demonstration
- 4:22Implementation and multi-agent LM
- 5:25Applications and user study
- 7:50Future research directions
- 8:32Conclusion and Q&A
related talks
18mShapeBots: Shape-changing Swarm Robots
18mZooids: Building Blocks for Swarm User Interfaces
25mDirecting a Swarm of Agents for Fun and Profit
10mMeet the Speakers
17mSwarmMap: Scaling Up Real-time Collaborative Visual SLAM at the Edge
09mSituated Live Programming for Human-Robot Collaboration
27mThe Shape of AI : How designers integrate AI into their workflow
07mHapticBots: Distributed Encountered-type Haptics for VR with Multiple Shape-changing Mobile Robots