Optimizing CephFS with Combining MDS QoS Scheduling and Static-Dynamic Subtree Partitioning
summary
Combines an external worker static partitioner (writes client traces to Elasticsearch, pins subtrees rarely) with a balance rank mask for the dynamic balancer and an mclock per-volume QoS scheduler (mclock library) to reduce metadata migration and tame noise neighbors. Evaluation uses vbbench, an in-house test framework (verDoge), 8 active MDS (12 GB MDS cache), workloads: 50k/500k/5M files; one link hit ~4M IOPS and caused long latency spikes; after static pinning plus link mask heavy subtrees migrate to ranks 4–7, dfetch count drops and MDS latency and throughput significantly improve; QoS enforces reservations/limits (e.g., 200 req/s aggregate). PRs exist and production rollout (backported to 14.2) is live.
AI-written summary · about
chapters (12)
- 0:00Introduction to project
- 1:22Line and Ceph infrastructure
- 4:29Workload analysis
- 6:48Addressing performance issues
- 8:47Subtree partitioning methods
- 11:52Combined partitioning approach
- 15:10Evaluation of the new approach
- 20:46MDS mclock QoS scheduler
- 24:03QoS evaluation and deployment
- 27:28Operational examples
- 29:48Future work and conclusions
- 32:26Q&A session
related talks
34mWhat are “caps”? (And Why Won’t my Client Drop Them?)
44mUserspace Assisted Scheduling via Sc...
01h 00mTrading at light speed: designing low latency systems in C++
39mPractical Container Scheduling: Juggling Optimizations, Guarantees, and Trade-Offs at Netflix
15mvBOIDs: Taming Chaos via Coarse-Grained Scheduling Abstraction for Containers
19mKVM/ARM at Scale: Improvements to the MMU in the Face of Hardware Growing Pains
48mRebuilding a RT Datastore: the design, deployment & performance story
10mLightning Talk: Resource Fairness and Utilization for Heterogeneous Batch/M...