Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots
summary
System predicts full UI trees from screenshots using three stages: an object detection model, an LSTM encoder plus stack‑pointer decoder that emits tree-building actions, and a group labeler that names intermediate nodes. Trained on AMP and Rico, it outperforms a heuristic baseline by up to 23% and retains high accuracy for screens up to 32 elements before degrading. Demonstrated applications include UI similarity search, accessibility metadata enhancement, and code generation from screenshots.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 0:29Problem motivation
- 1:54Screen parsing overview
- 2:38System architecture
- 4:21Visual demonstrations
- 5:52Evaluation and results
- 7:08Conclusion and future work
related talks
16mStateLens: A Reverse Engineering Solution for Making Existing Dynamic Touchscreens Accessible
06mRataplan: Resilient Automation of User Interface Actions with Multi-modal Proxies
29mCandy Crush QA AI Saga
20mRich Representations of Visual Content for Screen Reader Users
21mUsing Machine Learning To Improve UI/UX
26mHow to Cheat at Life with JS, CV, and AR
27mVisual UI Testing – Testing the Last Mile
01h 28mFrom RAG to Riches Building Enterprise Search Systems with LLMs