Hundreds of Terabytes of Data
summary
Pinterest grows from 1.7 TB/day in 2020 to the era of hundreds of TB/day and real deployments already run things like 625 TB on 200 nodes or 35×200-node clusters and a 12,800-core, 500 TB footprint. Known failures include a cluster-manager spike caused by cascading snapshot updates, shard-size skews that make half the nodes hot, and plugins that blow memory. Work shipped or underway: search backpressure, asynchronous and point-in-time search, segment-based replication, warm/cold tiers, cross-cluster and logical replication, and an SDK to make extensions safer.
AI-written summary · about
chapters (9)
- 0:00Start
- 4:09Compute, 2022
- 5:24Cluster Manager is a Bottleneck
- 7:29Shard Allocator is a Bottleneck
- 10:35Plugins are a Bottleneck
- 13:42Current Scaling Projects: Search
- 14:36Current Scaling Projects: Replication
- 16:08Current Scaling Projects: Storage
- 16:56Current Extensibility Projects
related talks
24mPinterest’s Exploration of Distributed SQL
34mBig and Small Data at @Twitter
54mProxySQL at Scale
23mZen: Pinterest's Graph Storage Service
48mBuilding Petabyte-Scale PostgreSQL Deployments
46mBig databases, many servers, data center or cloud - it's all here!
54mBreakout Session
19mBuilding KVStore for ML Workloads