Sharding: Growing Systems from Node-scale to Planet-scale
summary
Stripe runs a multi-petabyte cluster and the concrete advice is: extract a stable shard key, map it to shards, then iterate — start with read replicas (often absorbing 60–80% of traffic), split worker pools, use hash or range-based sharding for user/customer partitions, move heavy tables to separate databases, and evolve to cells and multi-region. Recommended tools include Vitess, CockroachDB, Yugabyte, and DynamoDB; avoid irreversible “trapdoor” decisions and prefer small, testable migrations.
AI-written summary · about
chapters (7)
- 0:00Introduction to sharding
- 0:25Strategy and best practices
- 6:24Why and when to shard
- 9:38Technical mechanics
- 16:20Architectural patterns
- 24:32The paved road
- 34:06Scaling and conclusion
related talks
32mSharding for mission-critical, internet-scale apps
52mTo Shard or Not to Shard
22mWhy Are Distributed Systems so Hard?
23mSharding for hyperscale and geographically distributed relational databases
37mScaling Vinted: A System That Never Stands Still
56mMastering Cloud Scalability and System Designs Deep dive
50mApproaches to implementing sharding in modern [No]SQL systems
30mOvercoming Distributed Databases Scaling Challenges with Tablets