PosePIE: Replace Your Keyboard and Mouse With AI-Driven Gesture Control
summary
PosePIE is an open‑source Python system that turns any webcam into a gesture input device using Ultralytics YOLO pose estimation with 17 COCO keypoints, TensorRT GPU optimization and ByteTrack for multiplayer tracking. It recognizes six core gestures (arm raise, lean, steer, jump, swipe, point), normalizes by shoulder width, uses time-based calculations and EWMA filtering, and maps gestures via a scripting API to keyboard, mouse (relative/absolute) or Xbox360 gamepad on Windows and Linux; demos include YouTube navigation and Trackmania steering.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 2:26PosePIE and inspiration
- 3:49Architecture and pose estimation
- 5:34Robust gesture detection
- 6:55Detected gestures overview
- 9:22Gesture detection techniques
- 11:07Input emulation and multiplayer
- 12:55Python scripting for gestures
- 14:24Live demos
- 15:52Future work and 3D pose
- 16:54Summary and open-source
- 17:39Q&A session part one
- 18:41Q&A session part two
- 19:45Q&A session part three
- 20:29Final Q&A
related talks
20mToward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines
10mProactive Sensing for Improving Hand Pose Estimation
28mBuilding Motion Controlled Websites in JavaScript
25mHands-free Coding with Gaze Control in JavaScript
55mMaking Every Game and App More Accessible with .NET and AI
21mHands-Free Coding With Gaze Control in JavaScript
20mNew User Interface - Part 1
57mApplied AI and Accessibility to Play Games in New Ways