Design of data-intensive applications
summary
Clear rule: pick the storage shape to match the workload and memory hierarchy not the buzzword. For high‑write workloads use LSM + Bloom filters with an append‑only log and background compaction; that design cuts seeks and reduces reads in practice. For low‑latency point reads use B-tree for point‑reads and in‑memory caches. Memory is orders of magnitude faster than disk (cache ~10x, RAM vs disk ~100x), so keeping hot data in RAM or using memory‑friendly layouts yields dramatic speedups. Examples mentioned: Cassandra/Influx style engines and log‑structured single‑file stores.
AI-written summary · about
chapters (13)
- 0:00Introduction to talk
- 1:09Data-intensive applications defined
- 3:21Memory and storage systems
- 8:14Java and memory constraints
- 11:14Random access vs hard drives
- 14:26Data structure optimization
- 21:28Log file optimizations
- 26:11Practical implementation tips
- 33:23Bloom filters explained
- 37:48Extending Bloom filters
- 42:14Database engine choices
- 45:20Performance benchmarks
- 47:49Q&A session
related talks
34mWhat We Talk About When We Talk About Disk IO
40mArchitecting for Data-Intensive Workloads
38mHow to run Real Time processing on Big Data
57mRead and write considered harmful
50mLet's talk about the cache!
01h 06mBest practices when accessing Big Data or any other data!
01h 14mRead and write considered harmful
26mColumnar Storage: Redefining Data Management for the Modern Era