improv BOF How to Scrape Data from any Website
summary
Concrete howtos and tradeoffs: a live curl + grep demo shows pulling meetup member lists, then HTML parsing with BeautifulSoup for quick one-offs, moving to Scrapy but hitting limits for async, proxy swapping and scaling, and using Selenium/Phantom (Chrome drivers) when sites render via JavaScript. Notes include scraping JSON APIs (CNN example), dedupe problems with Optimizely/NYTimes, cookie/session handling, and examples that sometimes work but often blow up at production scale.
AI-written summary · about
chapters (20)
- 0:00Web scraping challenges
- 1:49From curl to BeautifulSoup
- 2:11Scrapy strengths and limits
- 3:30Async scraping with Scrapy
- 5:23Crawler for model-based URLs
- 7:15Real scraping problems
- 8:49API scraping vs browser automation
- 10:18Making data accessible
- 11:22ML for data extraction
- 12:46Library usage poll
- 13:16Selenium for dynamic sites
- 14:41Hidden data behind UI
- 15:47Callbacks and session cookies
- 17:17Layout changes handling
- 18:20Duplicate content issues
- 19:05Scraping data from web sockets and APIs
- 20:38Quick scraping with browser tools
- 21:32Open-source scraping tools
- 22:16Google Sheets workaround
- 23:23Q&A
related talks
01h 01mLevelling Up Your Web Scraping Game
02h 51mGathering data from the web using Python
29mHigh Fidelity Web Crawling in Python
01h 01mWebcrawler
46mMastering Web Scraping: Unleash Your Data Extraction Wizardry!
39mMaster Web Scraping: Become a Data Extraction Ninja!
35mModern Web Data Extraction: Techniques, Tools, Legal and Ethical Considerations
32mWeb Scraping Best Practises