PROCESSING 40 TB OF CODE FROM ~10 MILLION PROJECTS WITH A DEDICATED SERVER AND GO FOR $100
summary
Cloned and analyzed roughly 10 million projects (about 15k short) totaling 40 TB of code, using shallow git clones, SCC for language/count/complexity, and a custom Go pipeline running on a cheap dedicated Hetzner box; cloning takes ~5 weeks, processing 1 TB of JSON took 49 hours single‑threaded on spinning disk, and overall cost targets hit near Go for $100 by avoiding AWS Lambda/Athena and using a 32‑worker Go program and tarred archives.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 2:07Analyzing 10 million projects
- 4:14Desktop and shell script attempt
- 6:20AWS Lambda and API Gateway
- 8:46SQS and S3 processing
- 10:34Dedicated servers and Taco Bell
- 13:21Porting to Go
- 14:44Efficient JSON processing
- 16:48Fun findings
- 18:53Lessons learned
- 21:01Dedicated servers power
- 21:43Legal implications
- 22:26Future plans
related talks
34m"CSI: Gopher"
44mAtomic Habits in writing high performing distributed code in the cloud
45mGitHub Archive - Processing and analyzing the billions of events generated by the w
01h 04mWhat can we learn from 750 billion GitHub events and 42 TB of code
22mOne Billion Row Challenge in Golang
30mHow Go Helped Accelerate Software Delivery at the BBC
53mGo at Mainframe Scale
01h 04mBuilding Efficient and Frugal Image Data Processing Pipelines