Beyond DOM: Leveraging Open-Source LLMs in an AI-Powered Appium Plugin
summary
An open‑source Appium plugin called find by AI uses multimodal LLMs (DeepSeek 32B, Q2, grok) plus omnipasser v2.0 via Hugging Face to convert natural‑language instructions into actions, returning bounding boxes, recalculating coordinates for device resolution, and issuing W3C actions to Appium. Experiments show roughly 70% accuracy today; visual detection handles cross‑platform UI drift and cosmetic changes but still mixes nearby icons and needs prompt tuning and polling, so it augments DOM locators rather than fully replacing them.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:05DOM-based element detection challenges
- 5:25Vision-based detection with LLMs
- 7:31Open-source vs commercial LLMs
- 11:07Vision-based Appium plugin
- 18:56Appium plugin integration
- 23:10Live demo: vision plugin
- 27:26Plugin code walkthrough
- 33:29Vision vs DOM detection
- 36:41Q&A
related talks
42mAppium: The Next Generation
42mHow AI can Assist Script-Based Automation
42mAppium: The Next Generation
46mAppium Native Application Crawler
57mAppium: Automation for Apps
31mUsing ML to make your UI tests more robust
33mExplore LLMs and Multimodal Approaches in Test Automation
52mAppium 2.0 - State of the Union