Hunting Heisenbugs
summary
Defines practical anti-heisenbug techniques: detect versus instrument, amplify rare events, and run targeted stress tests that make timing bugs happen reliably. Uses fleet math (a 10,000‑year bug shows up ~every 100 hours on 1M machines) and tools like rcutorture, kernel crash dumps, BPF and kdb, plus RCU near‑miss counter and CPU hotplug delays to inject races. Shows a focused test that drops MTBF from days or hours to ~12 minutes, and outlines combining orders‑of‑magnitude reductions to validate fixes.
AI-written summary · about
chapters (15)
- 0:00Introduction to Heisenbugs
- 1:22Detecting Heisenbugs in large fleets
- 7:22Why Heisenbugs are hard to debug
- 9:26Anti-Heisenbug techniques
- 10:09Increasing workload intensity
- 13:58Making bugs happen more often
- 17:48Modeling system behavior
- 25:57Counting near misses
- 27:23Swamping queues to reveal bugs
- 33:24Reproducing rare memory corruption
- 35:10Fast paths hiding bugs
- 37:58Combining rare events for testing
- 38:59Detecting errors before debugging
- 41:46Avoiding Heisenbugs entirely
- 42:49Q&A: Resources and techniques
related talks
39mAdventures in Bug Hunting
25mHunting Bugs While Sleeping
28mTesting Isn't Enough: Fighting Bugs with Hacks
39mA Common Taxonomy Of Bugs And How To Squash Them
21mHUSTEF 2014 - Marko Kuisma & Szilárd Széll
04mWhat Bug Hunting Taught Me About Hunting Bugs
44mTest Intelligence: Finding More Bugs in Less Time with AI
42mSurprise yourself with Property Based Testing