An Adventure in Race Conditions
summary
A proprietary image recompressor that handled ~620k slices/day and nine hours of work starts missing its SLA when input size doubles; initial per-stripe threads hit up to 1 second thread spawn cost and heavy oversubscription, plus subtle races: atomics used without mutexes break condition_variable wakeups, destroying the CV while tasks still run causes crashes, and early-exit locals are dangled. Fixes are moving the CV, requiring modification under the mutex, using a shared process context with weak_ptr→shared_ptr locking, and switching to a thread pool (16 stripes per machine) and futures (stlab) to reduce latency variance, though memory bandwidth becomes the ultimate limiter.
AI-written summary · about
chapters (9)
- 0:00Start
- 5:52Synchronization Primitives
- 6:13Undefined Behavior
- 10:40Beware of Your Compiler
- 16:39Thread Pool
- 31:35Error Handling
- 35:42Process Context
- 38:11Conclusion
- 58:06Strategy
related talks
07mRace Conditions in Details
44mRestartable Sequences: Scheduler-Aware Scaling of Memory Use on Many-Core Systems
15mEnemy of the State, Understanding the Many Flavors of Race Conditions
01h 02mMaking Games Start Fast: A Story About Concurrency
01h 04mDebugging data races
39mLet's talk concurrency
34mExploiting Race Conditions Using the Scheduler
30mScreaming Zombies and Other Tales: Race Condition Woes