Pinpoint Ceph Bottleneck Out of Cluster Behavior Mists
summary
Prototype collects per-request critical paths using distributed tracing and models costs with happens-before edges to show latency and throughput together. A 3 VM RBD image write test with 32 concurrent requests exposes a hotspot at objectmap and AIO steps which queue on OSTs and hit disk writes. An interactive Jupyter notebook ties costs to host/thread/request id so median latencies trace to the first OST queue occurrence. Tuning thread/shard counts, notably increased OSD shards removes queue bottleneck, resolves the bottleneck in the prototype.
AI-written summary · about
chapters (10)
- 0:00Introduction to talk
- 1:56Why performance matters
- 5:18Performance as latency and throughput
- 8:06Critical path analysis
- 11:02Identifying bottlenecks
- 14:21Proposed solution overview
- 15:48Demo: Tracing image writes
- 18:25Root cause analysis
- 20:58Interactive analysis capabilities
- 22:03Q&A and summary
related talks
38mEfficient Ceph Performance Troubleshooting in Production Using eBPF
39mUsing EBPF to Develop a Powerful Tool for Diagnosing Ceph Performance Issues
30mAbove the Clouds: Investigating Systems from 32,000 ft.
15mPerformance Management Q&A
22mReworking Observability In Ceph
01h 00mSystems @Scale Summer Q&A Week 4
37mMonitoring Service Architecture and Health with BPF
36m101 Ways to Crash Your Cluster