Chasing Bad Checksums: A Journey Through Ceph, TCMalloc, and the Linux Kernel
summary
Cluster runs dozens of nodes and hundreds of OSDs serving ~10 PB and sees about five checksum-crash OSDs per week; lab recreate fails every 3–4 days until adding heavy deletes which reproduces it every 3–4 hours. Root cause is tcmalloc’s madvise_free lazily freeing pages so a concurrent direct I/O write under memory reclaim can return a zeroed page, triggering checksum failures. Mitigations: add RocksDB retry, change madvise to DONTNEED in gperftools, and a kernel patch checking page refcount; upstream fix lands after months.
AI-written summary · about
chapters (11)
- 0:00Introduction and context
- 1:06Ceph cluster and issue overview
- 3:20Investigating checksum errors
- 5:57Reproducing the problem in lab
- 7:20Kernel and TCMalloc analysis
- 10:58Root cause in the kernel
- 13:44Direct I/O and memory reclaim
- 15:51The patch and fix
- 16:55Backports and status
- 17:43Madvise free in other projects
- 19:12Q&A
related talks
27mWhen Linux Memory Accounting Goes Wrong
45mIn-depth technical story: Fixing I/O performance for Windows guests in OpenStack Ceph clouds
19mMicroCeph: Simplifying Storage
52mWelcome & Opening Remarks & Meaningful Bounds Checking in the Linux Kernel
37mHardware Error Handling Improvement for Reliable KVM Hypervisor
36mTuning Memory and the Network Stack in Linux
28mExtreme OS Kernel Testing
31mEnhancing Scalability of the Vmalloc Mechanism in the Linux Kernel