Introducing Dat: If Git Were Designed For Big Data
summary
Dat is a prototype for data versioning and sync that treats tabular, large and real‑time feeds as a streaming realtime table replicator and aims for Dropbox‑style synchronization rather than Git cloning. It uses Node + LevelDB + npm modules to stream transforms and importers/exporters, hopes to avoid Git’s scaling limit (breaks around a million commits), and already runs as an open beta prototype funded by the Knight Foundation with modular transform tooling and live replication.
AI-written summary · about
chapters (4)
related talks
08mA Git-like Repository for Your Data Lake
26mCI/CD For Data – Building Data Development Environments With Open Source Stacks
24mMake data central feature
24mData Versioning Explained the Open Source Way
31mHow to Implement DataOps using
31mGetting ML Right in a Complex Data World
36mData Versioning - does it mean what we think it means?
32mDesigning Data Infrastructure in the Age of Generative AI