Rataplan: Resilient Automation of User Interface Actions with Multi-modal Proxies
summary
Rataplan records demonstrated GUI actions and links them to physical or keyboard proxies, using pixel-based reverse engineering and automated hover and click sequences to reproduce interactions. It resolves ambiguities by comparing nearby elements, uses OCR + proximity checks to handle cropped labels, auto-dismisses pop-ups by searching for “cancel/exit/later” or restarts the app if needed, and asks human-readable clarification questions to produce parametric queries; evaluations report significant improvements over Help, It Looks Confusing and Sikuli Slides.
AI-written summary · about
related talks
33mUsable PL: Human-Computer Interaction for PL Research
13mReMap: Lowering the Barrier to Help-Seeking with Multimodal Search
27mProcess Automation with TagUI
08mRapido: Prototyping Interactive AR Experiences through Programming by Demonstration
09mScreen Parsing: Towards Reverse Engineering of UI Models from Screenshots
10mApp-Based Task Shortcuts for Virtual Assistants
27mPower of Prototyping
48mIf I have to do this one more time, I quit!