Can You Move These Over There? Exploring an LLM-based VR Mover to Support Natural Multi-object Ma...
summary
A VR mover that combines spoken instructions, pointing and drawn lines with scene perception and an LLM to produce atomic API calls rather than generated code, achieving around 2 seconds response time (vs ~50s reported for prior code-generation systems). It supports asynchronous multi-object edits, area/line gestures, and an undo implemented by replaying prior atom sequences. A user study shows faster multi-object performance, less hand movement and reduces arm fatigue, with higher usability, lower NASA-TLX workload, and better UX than gizmos, hands, or rule-based voice.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 0:30What is object manipulation
- 1:34Problems with classical methods
- 2:16LM-based scene editing limits
- 2:59Natural interaction intuition
- 3:41Model and architecture
- 4:44Atomic function generation
- 5:05Interaction methods demo
- 6:53User study setup
- 7:36Technique comparisons
- 7:57User study results
- 8:59Exploratory interactions
- 9:42Conclusion and future work
- 10:04Q&A discussion
related talks
06mCoolMoves: User Motion Accentuation in Virtual Reality
45mLocomotion: Going places in VR
16mMagicalHands: Mid-Air Hand Gestures for Animating in VR
30mRoblox's Journey to Supporting Multimodality on vLLM
08mObject Manipulations in VR Show Task- and Object-Dependent Modulation of Motor Patterns
09mAdvanced Augmented Reality interactions
16mPortal-ble: Intuitive Free-hand Manipulation in Unbounded Smartphone-based Augmented Reality
19mOwnershift: Facilitating Overhead Interaction in Virtual Reality with an Ownership-Preserving ...