Can Applications Recover from fsync Failures?
summary
The study injects deterministic block failures on an unmodified kernel using dm-loki fault injector and a user-space tester cuttlefs user-space page cache and evaluates ext4, XFS, Btrfs and five apps (PostgreSQL, SQLite, Redis, LMDB, LevelDB). Key findings: all file systems mark dirty pages clean on fsync failure, ext4 data-mode reports errors late, Btrfs reverts page contents, and in-memory metadata can still point to non-overwritten blocks. Consequences include false failures, widespread data loss and corruption; simple retries fail, double-fsync helps, Btrfs is best for crash-restart but breaks rollback-based schemes.
AI-written summary · about
chapters (23)
- 0:00Intro
- 0:25How does data reach the disk?
- 1:22fsync is really important
- 2:10It's hard to get durability correct Applications find it difficult
- 2:42fsync can fail Durability gets harder to get right
- 3:31Why care about fsync failures? "About a year ago the PostgreSQL community discovered that fsync (on Linux and some BSD systems) may not work the way we always thought it is [sic], with possibly disastrous consequences for data durability/consistency (which is something the PostgreSQL community really values)."
- 4:03Our work Systematically understand fsync failures
- 4:41File System Results
- 5:30Application Results
- 6:27Outline
- 6:41File System | Methodology: Fault Injection
- 7:04File System Methodology: Workloads Common write patterns in applications • Reduced to simplest form
- 7:36File System Result #1: Clean Pages Dirty page is marked clean after fsync failure on all three file systems
- 8:47File System Result #22: Page Content File systems do not handle fsync errors uniformly • Page content depends on file system
- 9:52File System Result #3: In-memory state In-memory data structures are not entirely reverted
- 11:42Applications Five widely used applications
- 13:15Applications Results: Overview Ext4 Ordered Mode
- 13:50Crash/Restart Simple strategies fail Crash/restart is incorrect recovers wrong data from page cache • Example: PostgreSQL
- 14:53Applications Results #1: False Failures False Failures: Indicate failure but actually succeed
- 15:22Late Error Reporting All applications susceptible to data loss on ext4 data mode
- 16:20Btrfs winning?
- 17:34Applications Results Summary Simple strategies fail • Applications have moved away from retries
- 18:20Challenges and Directions
related talks
19mEvaluating File System Reliability on Solid State Drives
01h 12mRedundancy does not imply fault tolerance
41mFinding the Best Block Filesystem for Your Embedded Linux System
34mThe Computer Wants to Lose Your Data
18mOPTR: Order-Preserving Translation and Recovery Design for SSDs with a Standard Block Device Interface
42mJepsen 9: A Fsyncing Feeling
42mProduction FS: Adapt or Die
40mLinux Storage System Bottleneck for eMMC/UFS