(--We Steal--) Get Data from the Internet
summary
Practical techniques for extracting web data using Node.js tools: from small cheerio parsers to full browser automation with Puppeteer (including headless mode and browser pooling). Covers login-by-cookie, streaming large media downloads with token refresh, proxy+queue scaling patterns, and performance notes (prototype ~121 lines, dynamic pages ~+1s, ad‑blocking saves ~200ms); includes demos like a “send to TV” app and benchmarked code.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 2:20Parser projects
- 5:55No API sharing
- 7:18Puppeteer parser
- 10:10GitHub auth parsing
- 13:02Puppeteer demo
- 15:35Optimize with AdBlock
- 17:42Yandex Disk download
- 19:11Dynamic page parsing
- 23:56Browser speed
- 26:04Content protection
- 29:17Video and audio parsing
- 33:33Video links check
- 33:53Google and Yandex indexing
- 36:26Schema support
- 40:49Parser architecture
- 44:00Task queues
- 45:04Puppeteer alternatives
- 46:28Defending against parsing
- 51:03YouTube protection
- 55:05Conclusion
related talks
35mServer-Side Browsers: Exploring the Web’s Hidden Attack Surface
31mFinding Stealthy Bots in Javascript Hide and Seek
43mLive-Coding: Mastering Anti-Ban & Web Scraping with Scrapoxy
40mHostile JavaScript: Attacking and Defending the Browser
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!
43mLive-Coding: Become a Web Scraping Ninja with Scrapoxy!
27mReverse-engineering JavaScript to get rid of trust issues
27mReverse Engineering a Private API