I Made a Slow Thing Fast
summary
Identifies the real bottleneck with top, iostat and perftools.rb (pprof shows ~72% time in io_select), then applies three cheap fixes — reverse processing order, skip unnecessary fetches, and switch Faraday to net_http_persistent — cutting feedback time from ~15 minutes to ~2.5 minutes. Builds a persistent thread pool (Queues, ~100 workers) to reuse connections despite MRI’s global interpreter lock, reducing sync from ~1 minute to ~17 seconds, and the change works.
AI-written summary · about
chapters (29)
- 0:00Intro
- 0:52Overview
- 1:40The API
- 2:03XML
- 3:20API
- 3:48The Problem
- 5:07What is Slow
- 6:01Top
- 7:50Output
- 9:41ProofTools
- 10:28Generating Output
- 11:42Sync Process
- 11:59Failure Investigation
- 12:32Performance
- 13:10What is important
- 13:34Giving feedback
- 14:02Change document
- 14:13Reverse smell change
- 14:44Metadata changes
- 15:28HTTP persistent
- 16:25Benchmarks
- 18:11Global Interpreter Lock
- 19:24Whats Parallel
- 20:47Temporary Thread Pools
- 25:58The Loop
- 27:39Saving Standard Error
- 30:13Time
- 31:28Questions
- 32:09Threads
