ReMap: Lowering the Barrier to Help-Seeking with Multimodal Search
summary
ReMap is a prototype that embeds video help into desktop apps using speech-initiated queries and deictic resolution (clicking an element adds its accessibility label) with a server-side Web Speech API and five embedded video players. A 13-participant lab study shows it helps people multitask and stay focused while following videos; timeline markers let users jump by speech. 22% of spoken queries include deixis but only ~25% resolve correctly due to missing accessibility labels; ASR truncation and transcript distractions are unresolved issues.
AI-written summary · about
chapters (7)
- 0:00Introduction to ReMap
- 1:14Problem with help-seeking
- 2:42Introducing ReMap features
- 5:29User study with Canva
- 8:53Accessibility limitations
- 10:09Usability challenges
- 12:23Key takeaways
related talks
18mFacilitating Development of Pragmatic Competence through a Voice-driven Video Learning Interface
55mDigital Modernization Through Accessibility
16mLeaving the Reduced Reality and Into… AR, XR, VR?
06mRataplan: Resilient Automation of User Interface Actions with Multi-modal Proxies
23mDesigning for Accessibility
01h 00mInvolving People with Disabilities in UX Research
13mSupercharging user experience with machine learning
35mAccessibility in Action