Screaming Zombies and Other Tales: Race Condition Woes

on YouTube ↗30m

" summary

Four concrete postmortems of production race bugs: an S3 test bucket collision where concurrent test runs overwrite step data so smurf reads the wrong file; a transaction settler that races with voids and is fixed by marking "about to settle" plus a 60s settlement timeout to let web voids complete; an IoT case where heartbeats outrun sensor messages so stale timestamps drop updates and the team chooses not to rework the data model; and zombie reporters drowning metrics from leftover reporters that keep sending zeros to statsd, fixed by stopping those reporters.

AI-written summary · about

" chapters (6)

1. 0:00Introduction
2. 1:10What is incrementing
3. 5:55Story 1 Smurfs Law
4. 10:14Story 2 Transactions
5. 18:18Story 3 Fix
6. 22:40Story 4 Screaming Zombies

" related talks

• 01h 04mDebugging data racesCliff Click · Hydra 2020
• 30mA Series of Unfortunate BugsSatabdi Das · CppCon 2019
• 07mRace Conditions in DetailsYusef Mohamadi · GopherCon Europe 2023
• 37mCrashing BEAM applicationsGuilherme de Maio · ElixirConf EU Virtual 2020
• 34mExploiting Race Conditions Using the SchedulerJann Horn · Linux Security Summit Europe 2019
• 49mEverything is broken and that's Okay!Matthias Endler · localhost conference 2019
• 01h 03mAn Adventure in Race ConditionsFelix Petriconi · ACCU 2019
• 22mData race detectionNeven Miculinic · WebCamp Zagreb 2019