The New Generation Of Data Stores
summary
Cloud-first data stores move storage into S3 object storage and split compute so reads and writes scale independently while reducing cost, at the expense of higher median latency often measured in hundreds of milliseconds. Practical constraints include slow small-file access, the need to compact files and vacuum, local SSD caching to hide blob latency, and complex coordination for deletes and cache invalidation. Early examples are Aurora 2018, BigQuery, Redshift and Snowflake and recent moves by ClickHouse, Datadog Husky and projects like Nessie for point in time forks.
AI-written summary · about
chapters (15)
- 0:00Introduction and agenda
- 1:20Cloud shift and data stores
- 3:26Evolution of data store architectures
- 8:19Streaming and decoupling reads/writes
- 10:25Goals of modern data stores
- 11:51Hyperscalers driving change
- 13:56Scale to zero in databases
- 18:09Compute and storage separation
- 23:07Data warehouses leading trend
- 25:31Challenges of splitting storage/compute
- 27:58Licensing and open source shifts
- 30:05Development complexity increase
- 33:15Architecture and coordination needs
- 36:50Summary and future outlook
- 38:53Where to learn more
related talks
33mNext Gen of Datastores: Hot Queries, Cold Storage
28mThe architecture behind modern cloud data warehouses
29mAccelerate your cloud architectures with the right storage options
20mBuilding An Elastic Query Engine on Disaggregated Storage
20mNow Is The Time To Move To AWS Storage
43mSoftware Quality As It Relates To Data
54mPrimary storage for SaaS solutions
52mData storage for modern applications - Mike Lehan