It Is OK to Be Metastable
summary
Defines a metastable failure as a positive feedback loop that keeps a system degraded after a trigger and argues avoiding the metastable vulnerable state is impractical and costly. Presents measurements: cloud VM latency spikes up to ~3,000×, 99th percentiles of ~400 μs versus ~3.1 ms, and replicated‑state experiments on three VMs where an academic protocol drops throughput ~5× under added delay while production‑grade protocols mostly hold. Recommends three tactics: understand environments algorithms and workloads, trigger-resistant design, and protect vulnerable components with early load‑shedding.
AI-written summary · about
chapters (5)
- 0:00Understanding metastable failure
- 6:24Three pillars of stability
- 8:03Environment and algorithm design
- 23:15Trigger resistance strategies
- 33:21Protecting vulnerable components
related talks
25mWe're Still Down: A Metastable Failure Tale
37mWhen the Cure Is Worse than the Disease: Metastability in Recovery
34mDistributed Systems - What Can Go Wrong and How to Get It Right
29mThe Case for Limping-Hardware Tolerant Clouds
30mFast, Available, Catastrophically Failing? Safely Avoiding Behavioral Incidents in Complex Production Systems
17mOne-Size-Fits-None: Understanding and Enhancing Slow-Fault Tolerance in Modern...
29mHorrors of Distributed Systems
01h 01mLittle things and big things: lessons for building large systems.