Scraping LEGO for Fun: A Hacky Dive into Dynamic Data Extraction
summary
Shows an LLM-driven scraper that replaces fragile XPath/CSS logic with prompted visual element selection using Chromium hooked to GPT‑4o and frameworks like Firecrow/browser‑use; an old Scrapy/Playwright pipeline takes ~30 minutes, the LLM agent pulls ~200 LEGO products in ~3 minutes. Uses rotating IPs via a 5–10 phone cluster, mentions CAPTCHA solvers at ~$0.002 per solve and built LinkedIn automation that bulk‑messages/profiles leads; demos run, produce CSV/JSON and handle simple CAPTCHAs.
AI-written summary · about
chapters (8)
- 0:00Web scraping ethics
- 3:38Scraping evolution
- 8:33Agentic browsing with LLMs
- 12:06LEGO scraping demo
- 17:24Q&A on LLM scraping
- 26:40Proxy costs and setups
- 31:18Legal vs ethical scraping
- 33:29Live coding LLM tools
related talks
01h 01mLevelling Up Your Web Scraping Game
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!
45mThe Art of Web Scraping. Trees, Algorithms and JS Magic
26mWeb Scraping with Puppeteer and NodeJS
24mimprov BOF How to Scrape Data from any Website
19mHow to get clean web data for chatbots and LLMs
01h 10mTeach an LLM to Navigate the Web
39mMaster Web Scraping: Become a Data Extraction Ninja!