VizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World
summary
VizLens captures a photo, sends it to parallel crowd workers for crowd-assisted labeling, then uses SURF features, brute-force matching and RANSAC plus skin-color fingertip detection to localize the interface and speak what’s under a finger. It runs on an AWS GPU at ~8 fps with ~200 ms latency, crowd labeling takes ~8 minutes at $1.15 and 99.7% accuracy, and a user study with blind participants shows it supports real-time fingertip feedback and guidance across many static and some dynamic panels.
AI-written summary · about
chapters (9)
- 0:00Introduction to VizLens
- 1:52Challenges with prior solutions
- 3:18How VizLens works
- 6:15Computer vision for real-time feedback
- 9:00User study and results
- 12:52User feedback and improvements
- 14:37Limitations and future work
- 16:26VizLens version 2 and dynamic interfaces
- 19:36Summary and Q&A
related talks
16mStateLens: A Reverse Engineering Solution for Making Existing Dynamic Touchscreens Accessible
20mNew User Interface - Part 1
28mVigNET: An intelligent camera app that assists you...
11mVIP-Sim: A User-Centered Approach to Vision Impairment Simulation for Accessible Design
58mAccessible Spatial Computing
15mAccessibility: A line of code can be many things
45mBe My Eyes - Leveraging Generative AI to help the blind and visually impaired people
21mHands-Free Coding With Gaze Control in JavaScript