Real Cross-Platform Workflow UI Automation
Talkat JCON Europe 2023 · on Tuesday, 2023-06-20· at 10:00· Cinema 5, Im Mediapark 1, 50670 Köln, Germany·
summary
Live demo shows end-to-end cross-platform UI automation that actually works: a browser login to GitHub triggers a physical Android phone controlled over ADB control, the LastPass authenticator opens, an OCR extraction reads the TOTP and the code is typed back into the web form, completing 2‑factor authentication. Implementation uses computer‑vision-based visual selectors (bounding boxes) and relational selectors below labels rather than DOM/XPath, mentions Robot Framework and Playwright as orchestration layers.
AI-written summary · about
chapters (10)
- 0:00Introduction to UI automation
- 1:05Cross-platform UI challenges
- 4:38Native app automation limits
- 9:14Visual and layout testing tools
- 11:03Human-like automation goals
- 13:33AI vision for UI recognition
- 15:40Live demo of human-like automation
- 18:28Two-factor auth cross-platform demo
- 26:11Future of agent systems
- 29:48Closing and Q&A
related talks
36mReal Cross-Platform Workflow UI Automation
43mAutomating the real-user scenarios across multi-apps, & multi-devices
22mAutomating the difficult, real-world, multi-user scenarios
48mHow to Automate Everything
48mChallenges in UI Automation and How to Solve Them
29mAutomated Testing 101/Testing Web Apps 101
30mTesting Mobile Web Apps on Real Devices: How Fortune 100 companies do it
57mAutomating Android: Command line controls