Learnings & Challenges for the Integration of Touchless Interaction
summary
A German startup now part of the ballet Automotive Group develops hand-feature tracking that prioritizes fingertip positions, hand rotation, openness and handedness, rendering a live 3D mesh at ~60 fps for natural self-recognition. Prototype runs on mobile platforms, uses a headset-mounted depth sensor with tight calibration and an occlusion shader to place real hands in front of virtual objects. Demonstrated limits: fixed focal planes cause focus conflicts and current headset field of view forces close interactive objects to the size of a ping‑pong ball. Natural MR interaction works, but larger FOV and light‑field displays remain required.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 1:17Key requirements for mixed reality
- 2:43Superpowers and natural interaction
- 4:12UI interaction challenges
- 6:45Hand tracking and visualization
- 9:37Gesture recognition for interaction
- 11:06Hand occlusion in mixed reality
- 12:57Focus mismatch and light field tech
- 14:01Field of view limitations
- 15:45Technical requirements summary
- 16:05Conclusion and future outlook
related talks
17mFaceWidgets: Exploring Tangible Interaction on Face with Head-Mounted Displays
43mDesigning for Hands, Gestures, and VR
07mBringing back human interaction to a digital world
16mGetting hands on with VR
21mFaceTouch: Enabling Touch Interaction in Display Fixed UIs for Mobile Virtual Reality
09mRetargeted Self-Haptics for Increased Immersion in VR without Hand Instrumentation
32mNext generation human character control
23mFuture of User Interaction: Mixed Reality, AI and IoT