Understanding and Detecting Fail-Slow Hardware Failure Bugs in Cloud Systems
summary
Claims a systematic study of 48 real-world failures and finds fine-grained, fail-slow hardware behavior repeatedly breaks synchronized and timeout-protected I/O, causing indefinite blocking, silent failures, and data-corruption paths. Builds Sieve fault injection, a static analyzer that identifies synchronized and timeout-protected fault points, instruments hooks for fine-grained fail-slow injection, and uses grouping and context-sensitive exploration. Applies Sieve to ZooKeeper, Kafka, and HDFS, finds six new bugs (two confirmed) and reports efficiency improvements over prior injection tools.
AI-written summary · about
chapters (12)
- 0:00Fail-slow hardware failures
- 1:20Classifying cloud hardware failures
- 2:25Real-world impact of fail-slow bugs
- 3:30Fail-slow effects on detectors
- 4:1048 fail-slow failure cases
- 5:37Root causes of fail-slow failures
- 6:44Detection and fault injection methods
- 7:30SE fault injection framework
- 9:21Static four-point analyzer
- 10:27Timeout-protected regions
- 11:51Fine-grained fault injection with SE
- 12:54Conclusion and future work
related talks
29mThe Case for Limping-Hardware Tolerant Clouds
17mOne-Size-Fits-None: Understanding and Enhancing Slow-Fault Tolerance in Modern...
18mEfficient Exposure of Partial Failure Bugs in Distributed Systems with Inferred...
29mHorrors of Distributed Systems
31mWhat Breaks Our Systems: A Taxonomy of Black Swans
01h 04mDebugging data races
29mScaleCheck: A Single-Machine Approach for Discovering Scalability Bugs...
40mCross-System Interaction Failures: Don't Fail through the Cracks