Who's Afraid of Uncorrectable Bit Errors? Online Recovery of Flash Errors with Distributed Redundancy
summary
Proposes DIRECT — a set of techniques that let devices expose higher bit error rates by using existing replication and coordination: replicate metadata, recover minimal key ranges via RocksDB index blocks, and run repairs through the Paxos log for safety. Implements fixes in ZippyDB backed by RocksDB and HDFS, injects errors on 60 Facebook servers, and shows reduces application-visible errors >100x and recovery time >10,000x; HDFS tolerates 10k–100k× higher bit error rates.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:20Denser flash → shorter lifetime
- 0:55Shorter flash lifetimes are a problem . Datacenter operators must closely monitor flash writes
- 2:55Distributed error Isolation and RECovery Techniques (DIRECT)
- 4:03Bit errors in the storage stack
- 4:55Local data store errors: metadata
- 5:34Optimizing error recovery: strawman
- 7:34Implementations of DIRECT
- 7:58ZippyDB Overview
- 9:30RocksDB SST file layout
- 9:56Identifying corrupt data
- 10:34Index blocks identify corrupt range
- 11:18Safe recovery of corrupted data objects
- 13:03Error rate tolerance
- 14:05Size of key range + recovery latency
- 15:31Conclusion
related talks
01h 12mRedundancy does not imply fault tolerance
29mHorrors of Distributed Systems
36mDesigning for failure - architecting fault-tolerant system
19mSpeculative Recovery: Cheap, Highly Available Fault Tolerance with Disaggregated...
46mThe Do's and Don'ts of Error Handling
41mWhat can go wrong with the distributed system - the Hazelcast exp...
42mJepsen 9: A Fsyncing Feeling
40mResilient Segment Replication With Adaptive Refresh