Levelling Up Your Web Scraping Game
summary
Shows concrete choices and outcomes: prefer replaying HTTP requests over firing browsers, measures headless cost (≈250 meg RAM) and wraps puppeteer with a node web service instead of running it in PHP, uses guzzle and symphony http browser for PHP scraping, documents handling CSRF, cookies and headers, reverse engineers OAuth implicit flows and id tokens, implements pagination and edge-case checks, falls back to OCR and PDF tools (tesseract, qpdf, pdf to text) and mobile APIs when web UIs add captchas; demos run and code is on GitHub.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 1:38Overview
- 2:34Getting Data
- 5:35Headless Browser
- 6:10Performance
- 10:25Edge Cases
- 11:54Firefox vs Chrome
- 14:33Demo 1 Phone Networks
- 16:41Demo 2 Local Calling Guide
- 20:38Demo 2 Edge Cases
- 27:07Demo 3 Insurance Details
- 34:42Oauth Headers
- 38:17Parse URL
- 39:02Single Request
- 39:58JSON Back
- 40:08Recap
- 41:50Using a real browser
- 44:04Browser fingerprinting
- 48:03Antiscraping
- 48:46Mobile API
- 52:27OCR
- 52:55PDF Encryption
- 54:07Questions
related talks
24mimprov BOF How to Scrape Data from any Website
26mWeb Scraping with Puppeteer and NodeJS
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!
28mRethinking Browser Automation: Unleashing the Power of Puppeteer and Playwright
38mScraping LEGO for Fun: A Hacky Dive into Dynamic Data Extraction
45mThe Art of Web Scraping. Trees, Algorithms and JS Magic
29mHigh Fidelity Web Crawling in Python
01h 01mWebcrawler