I came for the easy concurrency I stayed for the easy composition
summary
A concrete program is described that checks whether 2 million sites use CloudFlare by NS lookup, implemented as 75 lines of Go, spawning 1000 goroutines and using channels plus sync WaitGroup. The claim: replacing slow xargs/dig pipelines with this Go pipeline drops runtime from ~18 hours to about 1.5 hours. The code generalizes via two interfaces (Factory and Task), is on GitHub, and it actually works.
AI-written summary · about
chapters (14)
- 0:00Start
- 2:04First thought
- 3:23Rough Architecture
- 3:56Quick type to encapsulate work and results
- 4:19Read stdin, stuff down channel
- 4:54Read results; write to stdout
- 6:12Easy concurrency
- 7:49factory and task
- 8:37Implement factory
- 8:47Implement task
- 11:24Starting State
- 12:05stdin empty; in closed
- 12:24Workers finish terminating
- 13:19Conclusion
related talks
46mSimplifying Scalable Cloud Application Development with Go
34mGo concurrency in the real world
24mMastering Concurrency in Go: From Patterns to Production
49mGet GOing with a New Language
22mOne Billion Row Challenge in Golang
10mGo I made all the mistakes
45mGo, Microservices and all the joy!
35mGo’s Trace Tooling and Concurrency