One-Size-Fits-None: Understanding and Enhancing Slow-Fault Tolerance in Modern...
summary
A systematic testing pipeline injects diverse slow faults into six modern distributed systems and sweeps ~100k fault combinations including 8000 configuration combinations; default settings trigger up to 98% degradation under some setups, while hand-tuned static configs can drop degradation to ~20% but overfit and break under different workloads. Scaling resources can amplify impact up to 10x and slow followers sometimes hurt more than leaders. A lightweight ADR library makes detection adaptive, finds faults within seconds, adds ~2.8% overhead, and yields up to a 90% reduction in impact.
AI-written summary · about
chapters (9)
- 0:00Fail slow faults in distributed systems
- 1:46Evaluating slow fault tolerance
- 4:14Automated slow fault injection
- 6:21Software configurations and slow tolerance
- 9:51Slow tolerance and deployment environments
- 10:53Slow fault attributes and system behavior
- 12:57Static thresholds in fault detection
- 14:40Adaptive detection for slow faults
- 16:07Summary and open questions
related talks
19mCAFault: Enhance Fault Injection Technique in Practical Distributed Systems via...
14mUnderstanding and Detecting Fail-Slow Hardware Failure Bugs in Cloud Systems
36mDesigning for failure - architecting fault-tolerant system
18mEfficient Exposure of Partial Failure Bugs in Distributed Systems with Inferred...
29mThe Case for Limping-Hardware Tolerant Clouds
01h 12mRedundancy does not imply fault tolerance
34mDistributed Systems - What Can Go Wrong and How to Get It Right
30mFast, Available, Catastrophically Failing? Safely Avoiding Behavioral Incidents in Complex Production Systems