Teach an LLM to Navigate the Web
summary
Claims a working browser agent built from scratch using Python Playwright and an LLM API and shows it actually navigates Amazon, adds a book to cart, and posts to LinkedIn while failing on captchas. Implementation details include converting DOM to an LLM-friendly schema, capturing screenshots for vision, caching indexes by URL, embedding the accessibility tree for robust locators, a clear observe-think-act loop, a small memory backed by SQLite, and about 18 tools (click, extract, tab ops); demos complete in a few steps and reveal practical limits.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:51Core architecture
- 8:19Planning and execution
- 11:26Designing tools
- 13:35Web scraping challenges
- 18:12Accessibility tree approach
- 23:55Document processing and memory
- 26:48Live demo
- 33:56Code deep dive
- 41:16Orchestrator flow
- 51:35Demo continuation
- 55:26Logs and optimization
- 56:31Security and sandboxing
- 1:00:19Handling ambiguity
- 1:03:10Evaluating performance
- 1:07:03Q&A
related talks
09mBuilding LLM Agents with Tool Use
25mWill LLMs really consume the Internet?
41mBuilding LLM Agents – Challenges and Learnings
47mAutomating Web Workflows with LLMs- 47mAutomating Web Workflows with LLMs
35mBuilding LLM-based Agents
17mWebLLM: A high-performance in-browser LLM Inference engine
39mLLMs everywhere: The future is now and the past keeps repeating itself