Building your own large scale web security scanning infrastructure in 40 minutes
summary
Repository delivers a working, open source web scanner implemented in Go (two‑month port) that uses DOM rendering with PhantomJS, HTML Simhash deduplication (95% threshold) to cut crawling time, a pub/sub queue with NSQ for horizontal scale, Elasticsearch/Kibana for aggregated results, and CI hooks for per‑commit scans. Current deployment runs ~2,500 scans/day with a goal toward 100,000, and an internal post‑processor reports ~98–99% XSS accuracy while reducing duplicate findings and site load.
AI-written summary · about
chapters (11)
- 0:00Why we chose Go
- 1:53Crawling infrastructure
- 7:16SimHash algorithm
- 14:44Experiments and results
- 16:52DOM crawling and rendering
- 20:37Inter-process communication
- 24:52Scalable architecture
- 30:20Running the distributed scanner
- 32:05Eliminating false positives
- 33:40Reporting and dashboarding
- 36:22Open sourcing the project
related talks
43mAround the Web in 80 Hours: Scalable Fingerprinting with Chromium Automation
32mAutomating Web Application Bug Hunting
50mScaling Security Assessment at the Speed of DevOps
58mOpen Source Pentesting and Security Analysis Tools: the DevOps way
39mAutomating Your Security Checks: Stop Hiding Your Flaws Next to the Monsters...
47mSecurity Gate: Building a Homegrown SAST Orchestration Platform
35mServer-Side Browsers: Exploring the Web’s Hidden Attack Surface
47mAutomated Security Scanning in Payment Industry