From datasets to tables in a multitenant data lake
summary
Salesforce moves from immutable dataset blobs to a table abstraction that hides hydration and schema drift, claiming simpler, reproducible access for many tenant-specific models. The API returns schema plus manifests via a scan, supports ever-growing table semantics, predicate-based manifest pruning and time travel, and enforces tenant isolation for per-customer models. Implementation experiments use Apache Iceberg for schema evolution, partition layout and versioning. Trade-off: producers pay more for alignment, consumers avoid manual stitching of snapshots (example: 1,001-record snapshot then tiny incremental sets of 4).
AI-written summary · about
chapters (7)
- 0:00Introduction and motivation
- 3:40Current challenges with datasets
- 7:38Salesforce multitenant data lake setup
- 12:23Limitations of datasets for consumers
- 17:25Introducing tables abstraction
- 21:06Why tables improve consumer experience
- 22:46Q&A session
related talks
28mBuilding A Governed Data Lake
22mThe Who, What, and Why of Data Lake Table Formats
25mBuild open format data lakehouse
29mServing a multi-platform Data Lake via Delta, Iceberg, Unity and Glue
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics
30mHarness the power of your data with the AWS Lake House Architecture
36mTable Formats Change Everything (By Not Changing Anything)
33mThe advent of the open data lake