Failing Better - When Not To Ceph and Lessons Learned
summary
Ceph excels at hyperscale but struggles in small or single-threaded deployments: a four-node POC often underperforms and offers little economy of scale, 20 TB fits better in one chassis than a distributed cluster, and very large single RADOS objects unbalance placement groups. Replication and erasure coding trade latency for correctness so recovery is slow across nodes, caches mask workload realities, and benchmarks mislead. Hardware monocultures cause systemic failures — e.g. monoculture NIC firmware panic — so use multi-vendor parts and size nodes to limit recovery impact; avoid front-end caching illusions and design for network latency.
AI-written summary · about
chapters (20)
- 0:00Introduction and setup
- 1:52When Ceph is the only choice
- 3:13Ceph design goals and trade-offs
- 6:07Centralization vs distribution in Ceph
- 9:38Recovery and network bottlenecks
- 12:05Recovery speed and data placement
- 15:37Starting with Ceph and small clusters
- 17:42Economy of scale with Ceph
- 19:25Big data and growth trajectory
- 21:13Workload considerations and scalability
- 23:40Gateways and latency impact
- 24:22Caching strategies and pitfalls
- 27:53Cache durability and availability models
- 29:36Benchmarking and latency expectations
- 31:51Hardware monocultures and risks
- 33:57Consumer hardware pitfalls
- 34:58Failure domains and rack design
- 36:21Node sizing and pricing models
- 38:08Ceph in public cloud environments
- 39:13Alternatives and closing thoughts