Live-Coding: Mastering Anti-Ban & Web Scraping with Scrapoxy
summary
Live-coding shows a working pipeline that uses Scrapoxy proxy waterfall + Scrapy and Playwright-driven browser farms to get 50 item pages past headers, IP rate limits and JS fingerprints, with dynamic spin-up/shutdown of cloud VMs to cut cost ~80%. Demo includes deobfuscating an encrypted anti-bot JS payload and replacing it with a forged submission so requests succeed without full browser rendering — practical tradeoffs and concrete commands, and a 1,900‑star GitHub project.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:07Isabella’s scraping challenge
- 5:03Trekyreview.com protections
- 6:27Introducing Scrapy
- 8:37Real-world scraping begins
- 10:45Bypassing user-agent checks
- 11:45IP rate limiting with proxies
- 13:30ISP versus residential proxies
- 15:41Managing proxies with Scrapoxy
- 19:17Switching to Lithuanian proxies
- 21:56JavaScript with Playwright
- 27:00Time zone alignment
- 29:05Decrypting anti-bot payloads
- 35:25Payload-mimicking spider
- 37:12Deobfuscation vs Playwright
- 37:35Q&A
related talks
16mLive-Coding: Mastering Anti-Ban & Web Scraping with Scrapoxy
43mLive-Coding: Become a Web Scraping Ninja with Scrapoxy!
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!
01h 30mMaster Advanced Anti-Ban & Web Scraping Techniques
50mArchitecture of an Anti-Bot Protection
34mArchitecture of an Anti-Bot Protection
44mDigital Cat-and-Mouse: Strategies to Outsmart Scrapers, Phishers, and Thieves
42mCracking the Code: Decoding Anti-Bot Systems!