When the Cure Is Worse than the Disease: Metastability in Recovery
summary
A global message bus called scribe (Kafka-like) experiences a cascade where issuing thousands of hourly backfills to fix missing partitions actually amplifies failures, creating a sustained positive feedback loop. Root causes are concrete: granularity mismatch (must reread full hourly partitions), thermal mismatch (cold storage hits instead of cache), and expectation mismatch (huge recovery bursts crash readers); ad hoc fixes—adding producer capacity, manual pacing and recovery shedding—reduce load but expose the need for bounded, priority-aware, incrementally recoverable replay paths.
AI-written summary · about
chapters (10)
- 0:00Metastability in recovery
- 1:01Scribe message bus context
- 4:57Data quality incident recovery
- 7:30Metastable failures and loops
- 13:28Cascading recovery failures
- 21:42Recovery amplifiers
- 28:23Ad-hoc solutions
- 30:33Recovery amplification strategies
- 33:20Resilience design considerations
- 35:48Key takeaways and risks
related talks
25mWe're Still Down: A Metastable Failure Tale
34mDistributed Systems - What Can Go Wrong and How to Get It Right
38mIt Is OK to Be Metastable
27mObserving Failure in Reactive Systems
23mDeciphering Cassandra’s Prophecy: Preventing the Fall of Your Application
57mReliability Nirvana
01h 12mRedundancy does not imply fault tolerance
42mArchitecting for Failures in Microservices