Rise Of The Machines - How Automated Processes Overtook the Web
summary
Akamai research presents concrete findings: a one‑day sample shows 85 billion requests with 8 billion bot requests (~9.5%), and a found mega‑scraper uses 3,000 IPs across 15 subnets where each IP issues <100 requests but the net generates millions of hits. Detection combines transactional signatures (UA/header quirks, phantomjs/selenium fingerprints, proxy/cloud IP ranges) with behavior profiling and botnet correlation, feeding a Big Data pipeline (Cloud Security Intelligence: 20 TB/day, 2 PB stored) and prefers management (rate‑limit, slow responses, serve stale data) over outright blocking.
AI-written summary · about
chapters (15)
- 0:00Web robots phenomenon
- 1:16Unexplained traffic spike
- 5:55Distributed mega scraper botnet
- 6:58Akamai’s cloud security
- 8:02Bot traffic volume and types
- 10:57Challenges posed by web bots
- 12:24Bot detection approaches
- 13:52Signature-based detection
- 16:51Bot platforms and tools
- 18:14Bot user agent mistakes
- 21:44Behavioral profiling for bots
- 26:28Distributed botnet detection
- 28:02Bot mitigation strategies
- 31:56Key findings and approaches
- 34:23Question and answer session
related talks
54mDetecting and managing bot activity more efficiently
49mAre you human or a damn bot?
20mHow to defend your sites from AI bots
01h 00mRock ‘Em Sock ‘Em Robots: Bot Swatting Like the Pros
23mBot or human? Detecting malicious bots with machine...
29mStandard Classification: Good Bot, Bad Bot: Characterizing Automated Brows...
48mWho’s in control: Human or Machine?
13mThe silent infrastructure crisis