From RUM to Robot Crawl Experience
summary
Monitoring often excludes robots; after enabling bot visibility, discovery shows 500,000–1,000,000 requests/day from Googlebot and similar crawlers, often 50–70% of traffic, with some teams running a separate data center or load balancer just for bots. About 20% execute JavaScript, affecting page render and performance. Google uses time to first byte and labels slow pages, so measure bot performance, coordinate with SEO, and apply bot-routing or throttling solutions.
AI-written summary · about
chapters (5)
related talks
50mHow We Crawl 60 Million Web Pages a Day
13mThe silent infrastructure crisis
41mSearch: State of the Union
41mAddressing the Challenges of Public Web Data
01h 00mRock ‘Em Sock ‘Em Robots: Bot Swatting Like the Pros
50mTechnical SEO for JS developers
33mDemystifying discoverable web apps
09mTammy Everts (SOASTA)