Billion Rows Walk Into a Go Bar
summary
Solves the 1BRC (one billion-line) input by profiling with pprof, iterating from a 2m27s prototype down to ~10s and cutting memory from gigabytes to tiny by specific changes: switch scanner.text→scanner.Bytes→custom byte parser, replace bytes.Split with index-based loop, batch lines and use buffered channels, match workers to CPU cores, tune buffer size (sweet spot ~1MB), use setGCPercent and unsafe string zero-copy to avoid huge allocations — concrete code edits and numbers instead of abstract advice.
AI-written summary · about
chapters (18)
- 0:00Introduction and speaker background
- 1:04Introducing the 1BRC challenge
- 2:28Why performance matters in development
- 3:31Initial profiling and unexpected findings
- 5:37Optimizing file reading with byte arrays
- 7:42Custom parser for temperature processing
- 9:48Reducing memory allocations in parsing
- 11:14Replacing built-in functions with loops
- 12:56Parallel processing and its pitfalls
- 14:20Batch reading and handling line breaks
- 16:29Buffer size trade-offs in reading
- 17:32Parallelizing with multiple workers
- 19:18External factors affecting performance
- 20:00Garbage collection tuning
- 21:04Zero-copy string conversion
- 22:30Key lessons from the challenge
- 23:12Further optimizations and resources
- 23:53Q&A session
related talks
22mOne Billion Row Challenge in Golang
40mTwo Go programs, three different profiling techniques, all in 40 minutes
29mOptimizing Go Code Without a Blindfold
14mOptimizing Go code without a blindfold
21mGo Right Ahead! Simple Hacks to Cut Memory Usage by 80&
21mUnderstanding Escape Analysis to Speed Up Your Code
26mPractical Memory Profiling
30mDynamically Instrumenting Go Programs