The Road to Intent-Driven Automation
summary
Argues that RPA shifts from “how” to what: users give an intent and an agent figures out UI steps. Demonstrates an experimental intent pilot that uses screenshots -> a multimodal LLM (GPT/Claude Sonet 3.5) + computer vision (Segment Anything and a proprietary detector trained on scraped pages) to generate pixel-accurate element locations and JSON actions, then executes them with a UI controller (PyAutoGUI). A recorded run succeeds (roughly 1 of 3), live demos often fail (around 1 in 10); core gaps are object localization, multi‑screen and Unicode typing.
AI-written summary · about
chapters (21)
- 0:00Intent-driven automation intro
- 1:06Computer vision AI tech
- 2:30Demo: Intent Pilot agent
- 4:38UI automation challenges
- 6:02Vision models for UI
- 8:30Visual Q&A in UI automation
- 11:27Why element localization matters
- 13:59Visual grounding for elements
- 16:13UI automation with prompts
- 17:38LLMs in planning role
- 19:04Agent prompt structure
- 20:30Comparison with Anthropic's Computer Use
- 21:54UI device controllers role
- 23:20Cross-platform UI controller reqs
- 25:26Agents as memory programs
- 26:32AI models as commodities
- 27:15Agents using tools
- 28:20Live demo: vision agent
- 33:28Testing agent with scrolling
- 34:54Vision agents today
- 36:17Q&A
related talks
43mThe Road to Intent-Driven Automation
08mThe RAod to intent-driven automation
48mHow to Automate Everything
24mThe truth about AI development
42mThe Path to Autonomous Testing
40mShould We Let AI Take Over Test Automation Completely?
46mIntelligence as Infrastructure: What Developers Must Learn to Build with AI
01h 17mThe year of the AI Agent: Beyond Automation