The Case for Limping-Hardware Tolerant Clouds
summary
Argues that “limping” hardware — disks, NICs, CPUs whose performance drops far below spec — is common and causes cascading failures: injecting a slow NIC at 0.1 Mbps or a limping disk makes Hadoop/HDFS/MapReduce workloads fail to trigger failover and speculative execution, saturating slots and causing a 13-node collapse (same effect at 60 nodes later). Methodology uses fault injection and white‑box tracing; a bug study finds 53 limping-related issues in ~500 reports. Recommends detection, quarantine or “fail rather than slow,” and cross-layer limp-tolerant designs.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:26Hardware Fails
- 1:11Facebook
- 2:08Is this hardware common
- 3:08In production
- 3:51Limping hardware
- 4:48Outline
- 5:35Methodology
- 6:22Example
- 7:42Results
- 8:38The Problem
- 11:10Work in Progress
- 14:40Buff Study
- 15:30Impacts
- 17:11Queue Management
- 17:58Discussion Conclusion
- 21:49Comments
related talks
14mUnderstanding and Detecting Fail-Slow Hardware Failure Bugs in Cloud Systems
17mOne-Size-Fits-None: Understanding and Enhancing Slow-Fault Tolerance in Modern...
36mDesigning for failure - architecting fault-tolerant system
16mPreview: Cloud Computing
38mIt Is OK to Be Metastable
53mResiliency and Availability Design Patterns for the Cloud
46mFault tolerance in Distributed Systems: A Case Study in Apache Spark
29mHorrors of Distributed Systems