Next Gen of Datastores: Hot Queries, Cold Storage
summary
Claim: moving index and transaction logs to S3-style blob stores makes datastores stateless and serverless, cuts replication (you write once not twice), and lets compute scale to zero; benchmarks report ingestion faster and much lower CPU because primary/replica coordination disappears. Tradeoffs: higher average latency but acceptable tail latency (Datadog’s Husky), provider limits and hot-spotting (GCP starts ~1k writes/5k reads/sec and requires warmup), S3-compatibility differences (an S3 repo analysis tool checks compatibility). Cost wins depend on batched write/read patterns.
AI-written summary · about
chapters (16)
- 0:00Next-gen datastores intro
- 1:04Managing datastores pain
- 3:49Serverless datastore promise
- 6:18Statelessness and S3 standard
- 8:47Cloud storage trade-offs
- 10:12Blob storage benefits and drawbacks
- 12:23Compute-storage split for scale
- 14:30Per-query cost and execution
- 15:57Cloud limits and hot spotting
- 18:47Avoiding hot spotting in blobs
- 20:10Serverless in practice
- 21:54Data tiers comparison
- 23:39Scaling reads vs writes
- 24:43Stateless datastore challenges
- 25:48Cloud native vs serverless
- 26:52Q&A on access patterns
related talks
41mThe New Generation Of Data Stores
20mBuilding An Elastic Query Engine on Disaggregated Storage
29mAccelerate your cloud architectures with the right storage options
32mLow Latency Serving of Offline Data: Efficient, Safe, and Reliable Data...
54mPrimary storage for SaaS solutions
23mState in a stateless world: data serving in the cloud
40mBeyond Relational: Data storage for modern applications
52mData storage for modern applications - Mike Lehan