Reimagining Mobile App Testing with Vision AI-Powered Agents
summary
Vision-language agents replace brittle locator tests (Espresso/Appium) by acting on pixels, using models like Quen, UI Venus and TARS, a command-specific prompt library, and a JPA-style genetic prompt tuner that evolves prompts from failures. CI/CD breakage rates cited ~30% drop with vision testing; element XY prediction sits around 40% XY accuracy today. A perceptual cache cuts action latency from ~5–7s to 2s, works on real devices and cross-app flows, but still misreads subtle visuals (wrong color selection, swipe offsets).
AI-written summary · about
chapters (13)
- 0:00Vision AI in mobile testing
- 1:04AI testing in mobile dev
- 3:09UI testing limits
- 6:41VLM processing screenshots
- 11:16Free from selectors
- 13:01VLM prompts for testing
- 15:26Prompt structuring
- 16:27Prompt refinement with JPA
- 18:31AI cross-app testing demo
- 20:19VLM caching optimization
- 21:43Prompt tuning lessons
- 22:47AI testing in CI/CD
- 23:08Q&A
related talks
30mHow Visual AI Makes Testing a Breeze
28mTesting at the Speed of Thought: QA in the Era of GenAI apps
44mYour Tests Lack Vision: Adding Eyes to Your Automation Framework
35mApp Testing with AI Tools – Make it Easy
45mMobile Visual Testing: Uphill Battle Of Mobile Visual Regression
11mDragonCrawl: Revolutionizing Mobile Testing with GenAI
18mExploration of tools, frameworks to take the pain out of mobile app testing!
43mTesting Agentic AI Applications: Beyond Traditional QA