Go concurrency in the real world
summary
Explains how channels, goroutines and sync.WaitGroup form a pipeline that parses gzipped VCF files, extracts population‑frequency fields, and writes to DynamoDB; the real program processes 270–280 million variants, runs 36 hours on a single 8‑core machine, and stays under 500 MB memory. Shows using configurable parallelism instead of buffered channels, select/time.After for timeouts, and context for cancellation, and reports the end‑to‑end pipeline works and scales by tuning worker counts.
AI-written summary · about
chapters (12)
- 0:00Introduction to Go concurrency
- 1:16Channels as blocking queues
- 4:05Goroutines and lightweight threads
- 7:35Waiting for goroutines to finish
- 9:20Synchronization with WaitGroup
- 11:48Building concurrent workers
- 18:08Real-world genomic data processing
- 25:36Beyond basic patterns
- 26:59Buffered channels and timeouts
- 29:30Select for channel operations
- 32:23Cancellation and cleanup
- 33:06Summary and Q&A
related talks
24mMastering Concurrency in Go: From Patterns to Production
16mIn the land of Gophers from age 3
15mI came for the easy concurrency I stayed for the easy composition
39mLet's talk concurrency
18mApplied concurrency in Go
22mOne Billion Row Challenge in Golang
09mConcurrency for beginners
01h 27mFlash Talks