~/talk

Sustainable scrapers

on YouTube 53m
Talkat PyData DC 2016 · 2016-10-07..10-09· [developer-experience]

speaker

name:David Eads
links:none on file

resources

video:https://youtu.be/ECM-a8Y_OjY
slides:none on file

summary

NPR builds a scraper that actually produces structured data about gun sale listings on Armslist.com and shows the engineering tradeoffs: Python parsing with BeautifulSoup, parallel workers driven by GNU parallel, and an EC2 C3.8xlarge instance that harvests roughly 880,000 URLs in about 16 minutes for around $0.50. The project documents URL patterns, handles odd DOM edge cases, skips tests under deadline, wrestles with terms of service and ethics, and uses throttling and headers so the scraper works without taking the site down.

AI-written summary · about

chapters (11)

  1. 0:00Sustainable scrapers intro
  2. 2:09Not about guns or Armslist
  3. 9:51When to use a scraper
  4. 13:42Ethics and legality for scrapers
  5. 21:08Beyond data collection
  6. 29:39Scraper architecture
  7. 41:43Model to script
  8. 44:12Parallelization with GNU Parallel
  9. 47:05Cloud for scraping
  10. 50:15Responsible scraping
  11. 52:20Publishing results

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.