Lip-Interact: Improving Mobile Device Interaction with Silent Speech Commands
summary
A smartphone front camera plus an end-to-end deep learning pipeline recognizes mouth movements to trigger apps and system actions, implemented first with a connected PC and then on-device. The prototype supports 44 silent commands, gathered from a user elicitation, trained on data from 22 participants (≈1,000 augmented samples), and achieves 95–99% recognition depending on validation and updates. Demo actions include screenshots, app launches, home/back, and showing payment codes; limitations include visible-mouth requirement, latency and edge-case misrecognitions.
AI-written summary · about
chapters (5)
- 0:00Start
- 5:36Design Constraints
- 10:03Spatial Transformer Network
- 12:05Training and Evaluation
- 13:16Demo
related talks
06mSensing to Hear: Speech Enhancement for Mobile Devices Using Acoustic Signals
36mVoice Enabling Mobile Apps and Custom Hardware Devices
35mHaving a Voice on Android: How hard can it be?
43mThe State of Voice Interactions on Android
30mImproving Voice Command Recognition Accuracy Using DialogFlow NLP
54mHarnessing On-Device AI to Transform the Smartphone Experience
34mVoice Search Using NLP & CoreML
58mVoice Control Made Simple – Panel Discussion