Laziness, logic and slave trade in the realities of scraping stores
summary
How to scrape a million different online stores without large teams of annotators or a fancy AI: practical techniques, engineering trade-offs, and where to rely on simplicity when algorithmic complexity matters. The talk covers concrete strategies the presenter’s team used (including replacing recursive approaches) to make large-scale web parsing robust and maintainable.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:00Millions of sites parsing
- 8:00Lazy patterns hypothesis
- 12:19Page markup structure
- 19:28Pattern discovery in markup
- 23:04Performance optimization
- 27:01Recursive traversal algorithm
- 33:01Quality assessment and errors
- 37:17Existing solutions limits
- 40:09Data postprocessing cleanup
- 41:35Data refresh frequency
- 42:38Q&A
related talks
40mAvoid the top 5 web data pitfalls when developing AI models
24mimprov BOF How to Scrape Data from any Website
32mWeb Scraping Best Practises
02h 51mGathering data from the web using Python
50mPerformance tricks learned from low level developers
28mData Harvest: Unlocking Insights with Python Web Scraping
35mLittle Data Soup with Spinach (Web Scraping)
29mDive into Scrapy