How We Crawl 60 Million Web Pages a Day
summary
A team built a web-crawling content system for a search engine that visits 14 million domains and fetches up to 100 pages per domain per month, then re-crawls monthly. The system enforces politeness (respecting robots.txt and limited RPS), targets at least 600 pages/sec throughput, and is designed for minimal hardware and operational costs.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 3:19System requirements
- 7:36Architecture evolution
- 11:16First crawler version
- 13:50Priority queue
- 18:31Duplicate handling
- 22:02Final architecture
- 24:54Problems and solutions
- 31:40Current infrastructure
- 32:01Key findings
- 37:41Metrics and monitoring
- 41:04Content processing
- 43:58Language and framework choice
- 45:04Robots.txt and host distribution
- 46:06Connection handling
- 48:21Future development
related talks
41mAddressing the Challenges of Public Web Data
06mFrom RUM to Robot Crawl Experience
50mTechnical SEO for JS developers
34mHow We Handle Over 1 Trillion HTTP Requests ...
17mSprinter: Speeding Up High-Fidelity Crawling of the Modern Web
13mThe silent infrastructure crisis
56mHow AI Ate My Website
41mSearch: State of the Union