The Story of How We Wrote a Crawler on Outsource
summary
A case study about outsourcing development of a web crawler: what problem required the crawler, how the team organized the outsourced work, and practical details of building and integrating the crawler in production. Covers technologies and processes used for implementation, testing, and delivery while working with external contractors.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:33Web scraping vs crawling
- 4:23Scraper vs crawler
- 6:10Tool selection
- 10:29Task setup
- 13:41First results
- 16:58Performance optimization
- 18:49Distributed architecture
- 20:59Version comparison
- 23:11Client testing
- 25:07Client types
- 26:53Early communication
- 27:57Development takeaways
- 28:20Conclusion & Q&A
related talks
36mDevOps in Outsource… Really!
26mMaking internets great again - a story of working with legacy code
52mSmall Steps Are the Fastest Way Forward: Life Beyond Agile & Scrum
23mOur Journey from in-House CD System to Open Source
35mScaling an Open Source CI with a Remote Team, the Story behind Travis CI
43mLive-Coding: Mastering Anti-Ban & Web Scraping with Scrapoxy
54mEffective SCRUM in Distributed Teams
49mHow We Build the Web in Skype