Data Checking at Dropbox
summary
Dropbox runs periodic sanity checks that scan every storage unit (e.g., hundreds of thousands of hard drives) and re-read blocks to verify checksums, validate filesystem logs, and confirm metadata matches block contents. The system partitions data, assigns a run with durable cursors to distributed check runners, and reports per-run violations plus cursor progress and historical comparison. When violations appear an auto-remediation script can repair partitions but must support backups, retries and post-repair rechecks to avoid making corruption worse; inject real corruptions to verify the checkers themselves.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:47Agenda
- 2:35Data Passing Through
- 2:52Durability Incident
- 4:24Block Scrubber
- 5:22Block Storage
- 5:58Looping
- 6:21Filesystem Verifier
- 6:57Filesystem Checks
- 7:48System Check Pattern
- 8:43Database Trust
- 9:25Data Model
- 9:50Checks
- 13:22Reporting
- 14:44Remediation
- 15:53Check the checkers
related talks
40mThe Mysteries of Dropbox
52mData Corruption Bugs: Diagnosis and Lessons
19mEvaluating File System Reliability on Solid State Drives
20mDebugging Data Corruption in PostgreSQL
30mCluster storage systems gotta have HeART...
53mA DBA's Guide to the Proper Handling of Corruption
01h 12mRedundancy does not imply fault tolerance
29mThings might go wrong in a data-intensive application