Affordable NVMe Performance on Ceph & Ceph on NVMe - True...
summary
Ceph on NVMe works if planned: expect ~0.7–0.8 ms 4K write latency with replica 3 and tuned CPUs, but vendor whitepapers often use replica‑1/2, top‑end CPUs and exotic NICs to hit inflated IOPS. Benchmark with single-thread 4k rados bench, warm up ≥10 minutes, repeat ≥4 runs and drop the first. Disable deep C‑states, trim OSD logging (–10–15% latency), allow 8–12 GB RAM per OSD, avoid caching drives in front of NVMe, and size OSDs ~4 TB for best CPU/IO balance.
AI-written summary · about
chapters (21)
- 0:00Introduction to talk
- 1:25Is Ceph right for you
- 3:10Planning ahead for Ceph
- 5:17Defining KPIs for Ceph
- 8:08Talking to vendors and benchmarks
- 10:55Networking and replication trade-offs
- 12:18Avoiding vendor pitfalls
- 14:49Performance vs safety trade-offs
- 16:42Verifying vendor claims
- 19:06Benchmarking with fio
- 21:34Preconditioning RBD images
- 23:06Networking and CPU tuning
- 25:31CPU tuning for latency
- 26:55Disabling OSD logging
- 27:37Tweaking OSD op threads
- 29:29Upgrading to Mimic
- 30:59Choosing the right CPU
- 32:25Increasing placement groups
- 33:56Sample affordable systems
- 36:10Performance benchmarks
- 39:27Q&A session