Yandex Alice: music control with gestures
summary
Built a gesture music controller that beats Google MediaPipe on precision and range by combining a hand detector and a separate lightweight classifier. Collected labeled data from 5,000 participants, mined extra images via visual search, Stable Diffusion skeleton generation, and crowdsourced classification, then trained MobileNetV2+SSD initially and switched to a quantization-friendly YOLOv8 detector. The classifier runs at 6 ms on-device, system achieves ~3.5 m range with lower false positives than MediaPipe.
AI-written summary · about
chapters (9)
- 0:00Yandex Alice introduction
- 1:32Music gestures feature
- 2:37Data collection challenges
- 4:26Surpassing MediaPipe goals
- 7:23Gesture recognition mining
- 12:53Detector and classifier fusion
- 14:24Model results comparison
- 15:32Project lessons
- 16:37Q&A
related talks
32mAdvanced on-device ML made easy
21mPosePIE: Replace Your Keyboard and Mouse With AI-Driven Gesture Control
49mImage Recognition, Object Detection, Model Evaluation
20mToward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines
15mMediaPipe Web: Bringing cross-platform AI tech to the browser
19mHand gesture-controlled robot through the Internet
45mWhat if AI were the solution to understanding sign language?
25mAlternative Smartphone Interactions based on Sensor Data