Bots & Humans: Maintaining AI agent and automation in an AI-bot web
summary
Claims the web is increasingly closed, citing 199 million requests to docs from Mentify and Cloudflare blocking 400 billion bots, then shows a build: uses browser automation (Selenium/Playwright), proxies, CAPTCHA solvers and TLS-fingerprint workarounds to produce an instant API from a single URL, demoing extraction from Coolblue.nl into JSON. Scales to millions, reports cost and blocking limits, and ships self-healing scrapers that detect DOM breakage and patch automatically so the pipeline stays working.
AI-written summary · about
chapters (7)
- 0:00Web data trends
- 4:57Closed web and bot access
- 9:28Large-scale scraping challenges
- 11:12Instant API generation demo
- 19:03Scraper maintenance and scalability
- 22:13Self-healing scrapers
- 23:16Call to action
related talks
19mHow to get clean web data for chatbots and LLMs
53mAI Didn’t Kill the Web, It Moved in!
20mHow to defend your sites from AI bots
21mPower your AI Agents with serverless browsers
49mMagic of Automation and Everyday Chores
29mStandard Classification: Good Bot, Bad Bot: Characterizing Automated Brows...
26mMocking and Deploying an AI Chatbot in Minutes
49mAre you human or a damn bot?