Why We Have 17 Clusters Instead of One (and Why You Probably Shouldn’t Do Th...
summary
CDN77 splits storage into 17 clusters (14 HDD, 3 SSD) behind an Nginx load balancer using an nginx chash module that hashes the first four chars of CRC32 from client object names to pick a cluster, then round‑robins across RGW ports for node-level distribution. Design choices: replication (not EC), two locations for DR, heavy read optimization. Tradeoffs: major operational overhead (user/reshard ops on 17 clusters), painful bucket listing across clusters, a wrong bucket‑name length forced re‑uploads, and a lifecycle job triggered a 2am power spike.
AI-written summary · about
chapters (13)
- 0:00Introduction and team intro
- 0:22Talk structure overview
- 1:07Problem and storage context
- 2:12Solution requirements and constraints
- 2:53Final solution design
- 3:57Consistent hashing implementation
- 5:40Topology and load balancing
- 6:47Hardware and networking setup
- 7:50Management overhead challenges
- 9:38Scaling and adding clusters
- 10:42Real-world usage lessons
- 12:26Outage and performance insights
- 13:09Q&A disaster recovery
related talks
33mWhy Are Distributed Systems So Hard?
15mFrom Chaos to Clarity: Deciphering Cache Inconsistencies in a Distributed...
30mUnderstanding Bottlenecks & Conquering them
22mWhy Are Distributed Systems so Hard?
24mAverting crisis at multi-petabyte scale
37mSharding: Growing Systems from Node-scale to Planet-scale
06mNobody Knows What PGs are Good For, Only I Do
32mStretch Clusters in Ceph: Algorithms, Use Cases, and Improvements