Confessions of a Professional Data Hoarder
summary
Government and medical systems must keep every historic record searchable with exact timestamps, so the design records material time vs formal time (and corrected times) and treats all updates as append-only with pointers rather than overwrites. Practical numbers and tradeoffs appear: hobby hoarders hit 12 petabytes, tapes and racks cost tens of thousands, drives draw watts, and denormalized schemas plus indexes or event streams (Kafka, Neo4j, Cassandra) are used to make historical queries performant and auditable.
AI-written summary · about
chapters (10)
- 0:00Data hoarding origins
- 4:07Hoarding journey and costs
- 8:25Data hoarding as a problem
- 10:08Professional hoarding areas
- 13:22Government storage and compliance
- 19:41Time in databases
- 26:20Mistakes and corrections in data
- 33:46Restoring historical data
- 39:04Querying historical data
- 41:52Q&A
related talks
30mCatching Waves with Time-Series Data
50mKeeping historical data in tables forever – mission (im)possible!
01h 00mA Brief History of Data Storage
46mBig Data. Or Just Too Much Data?
01h 19mThe Future of the Past: Challenges in Archival Storage
26mBig Data Is Modern Data
52mData storage for modern applications - Mike Lehan
01h 05mA Brief History of Data Storage