Growing spiders to crawl the web
summary
Practical guide to building PHP web crawlers that actually scale and stay unblocked: use curl_multi reuse to keep sockets open instead of opening thousands of TCP connections, push work through queues (gearman) to avoid IO waits, parse with XPath not brittle regexes, and run JavaScript pages with PhantomJS. Measure everything with statsd, alert on spikes, rotate proxies and cookies to avoid blocks, and hit production rates like 100 million pages per server per day by separating download, parse, and store stages.
AI-written summary · about
chapters (5)
related talks
29mHigh Fidelity Web Crawling in Python
01h 01mWebcrawler
01h 01mLevelling Up Your Web Scraping Game
24mimprov BOF How to Scrape Data from any Website
29mDive into Scrapy
39mMaster Web Scraping: Become a Data Extraction Ninja!
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!
24mBots & Humans: Maintaining AI agent and automation in an AI-bot web