The silent infrastructure crisis
summary
Fastly’s Q2 2025 analysis claims visibility into 6.5 trillion monthly requests, finds AI bots comprise ~50% of traffic (Imper), and shows fetcher spikes up to 39,000 requests per minute that force autoscaling, higher CDN bills, and polluted analytics. Meta supplies ~52% of crawler volume while OpenAI supplies ~98% of real‑time fetches, so rate limits, IP verification, robots.txt/RSL signals, proof‑of‑work and advanced bot management become concrete engineering controls rather than theory.
AI-written summary · about
chapters (9)
- 0:00AI bots and web performance
- 1:51AI bot traffic types and impact
- 3:59AI bot scraping costs
- 6:48AI bot verification and trust
- 8:11Fetcher spikes and infrastructure strain
- 9:57Geographical and industry risks
- 10:39Bot management strategies
- 12:01Action plan for AI bot traffic
- 12:45Future outlook and advantage
related talks
38mRise Of The Machines - How Automated Processes Overtook the Web
54mDetecting and managing bot activity more efficiently
20mThe AI Threat to Content Creators and the Web and What We Can Do About It
06mFrom RUM to Robot Crawl Experience
20mHow to defend your sites from AI bots
49mBuilding Fastly's Platform for Scale: Continuous and Incremental
36mDead Internet Theory: Fiction, or Future?
34mHow We Handle Over 1 Trillion HTTP Requests ...