Around the Web in 80 Hours: Scalable Fingerprinting with Chromium Automation
summary
Built a distributed scanner that runs real Chromium browsers via Chromium remote debugger, automated in Go, coordinates with NSQ, writes blobs to S3 and metadata to Postgres, and shreds tables and files to scale. It fingerprints 1,000,000 Alexa sites, captures 72 million responses and ~2 million JS library/version detections, finishes a full run in 15 hours for about $343, exposes 722 unique CSP violations, and survives Chrome crashes with browser pool, killwatch and batching/upsert import tactics.
AI-written summary · about
chapters (49)
- 0:00Introduction
- 0:30Who am I
- 1:05Background
- 1:28Scanners
- 5:06NSQ
- 5:36Help Features
- 7:05Visual Walkthrough
- 7:28Creepy Crawler
- 7:38Automation Requirements
- 8:43Chrome
- 9:19Application Logic
- 10:18Report
- 11:18Go Templates
- 12:05Automation Tool
- 12:29CSP Events
- 13:41JavaScript Libraries
- 14:36JavaScript Queries
- 16:17DOM Content Loaded
- 17:41Contamination
- 18:14Pooling Browsers
- 18:51Chrome Bug
- 19:06Solution
- 19:32Crash Safaricom
- 19:49History Push Bug
- 20:06KillFace
- 20:59Challenges
- 22:59Storage
- 25:17Database Terminology
- 26:27Network Traffic
- 26:51Lock Intentions
- 27:40Buffered Channels
- 28:16Temporary Tables
- 30:16Large Tables
- 30:46Table Sharding
- 31:23No Table Sharding
- 31:56Creating a Shard Key
- 32:30Limitations of Sharding
- 33:58S3 Key Value Store
- 35:34Example
- 35:53Right Ahead Log
- 36:43Statistics
- 37:40CSP Violations
- 38:21Script Source Violations
- 38:44jQuery
- 39:04Angularjs
- 39:56JavaScript
- 41:04Insight Guides
- 41:38Summary
- 42:35Thanks
related talks
39mBuilding your own large scale web security scanning infrastructure in 40 minutes
35mServer-Side Browsers: Exploring the Web’s Hidden Attack Surface
48mA Weakness In Chrome & A New Hacking Technique
28mRethinking Browser Automation: Unleashing the Power of Puppeteer and Playwright
40mAn Analysis of the Fingerprinting-Scripts and their Actors on the Internet
41mChrome Security Secret Sauce
45mThe evil friend in your browser
21mFp-Scanner: The Privacy Implications of Browser Fingerprint Inconsistencies