Making Ceph Fast in the Face of Failure
summary
Ceph adds concrete controls and algorithms to keep client I/O responsive during recovery: default OSD recovery sleep = 0.1s (HDD), 0s (SSD), 0.025s (hybrid) tuned from BlueStore + FIO tests; Luminous hits >90% of baseline IOPS (Hammer <10%). Mimic adds async recovery using PG-log delta (min entries 100) to postpone slow OSDs; Nautilus enforces a hard PG-log cap, replaces log-only selection with a min cost (log delta + missing objects), backports limits, adds pool-level force recovery, an experimental erasure-code plugin, and partial recovery — experiments show throughput stays near baseline.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:46Log Based Recovery
- 4:06The Dark Ages
- 5:51Infernalis
- 6:06Luminous
- 6:56Experiments
- 9:28Aluminous
- 9:42Forced Recovery
- 10:32Mimic
- 12:07When to Perform a Sink Recovery
- 15:39Sync Recovery Experiments
- 17:05Time Series Graph
- 17:39Hard Limit for PG Log Length
- 19:06Forced Recovery and Backfill
- 20:09Improved Performance
- 21:29Improved Backfill
- 22:22Recovery Sleep
- 23:14Code Plugin
- 30:00Summary
related talks
44mCeph QoS Refinements for Background Operations Using MClock
38mEfficient Ceph Performance Troubleshooting in Production Using eBPF
49mIntro to Ceph, the Distributed Storage System
02h 59mCeph Storage: a Caffeinated Primer
34mModernizing Ceph Deployments at CERN: CephFS and Object Storage Across Data Centres
12mCeph QoS: How to support QoS in distributed storage system
29mCeph Operations at Scale
38mTesting Ceph: Pains & Pleasures