IASO: A Fail-Slow Detection and Mitigation Framework for Distributed Storage Services
Talkat USENIX ATC '19 · on Wednesday, 2019-07-10· at 12:20· USENIX ATC Track I: Grand Ballroom I–VI·
summary
System detects and isolates “fail‑slow” nodes using AIMD-inspired scoring on per‑peer timeout ratios, publishing per‑minute scores to a score DB and taking the 30th‑percentile over a 10‑minute sliding window, then clustering with DBSCAN to flag outliers. Deployed 1.5+ years across 39,000 nodes, it finds 232 incidents in a 7‑month study (1.02% annual rate), reports 96% detection accuracy with a 3.7% false‑positive rate, and applies mitigations like revoke leadership leases or shutdown VM/node to stop cascading outages.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:28Fail Slow Faults
- 1:23Fail Slow Example
- 3:02Fail-slow severity
- 3:24Fail-slow problem space
- 5:27IASO Architecture / Solution
- 7:11Detection: Score characteristics
- 8:00Detection: First unsuccessful attempt
- 8:44Detection: Second unsuccessful attempt
- 9:50Detection: Final algorithm
- 12:55Detection: Score Analysis
- 14:27IASO in Production
- 15:22Mitigation of Fail-slow Node
- 17:05Resolution of Fail-slow failure
- 17:39Conclusion
related talks
14mUnderstanding and Detecting Fail-Slow Hardware Failure Bugs in Cloud Systems
17mOne-Size-Fits-None: Understanding and Enhancing Slow-Fault Tolerance in Modern...
29mThe Case for Limping-Hardware Tolerant Clouds
54mDistributed Caching in Real-Time Payment Systems
15mFiDe: Reliable and Fast Crash Failure Detection to Boost Datacenter Coordination
27mReducing Client Latency and Timeouts When Running Cassandra on Public Cloud
46mHow Cheap Learning (not deep learning) Finds Faults in Distributed Systems
13mAccept Partial Failures, Minimize Service Loss