DuckLake: Learning from Cloud Data Warehouses to Build a Robust “Lakehouse”

on YouTube ↗55m

" summary

DuckLake stores metadata in a SQL catalog so file pruning runs as SQL queries against ducklake_file and ducklake_column_stats instead of traversing JSON manifests, giving provable pruning and snapshot isolation via database transactions and inline writes for fresh data. Motherduck implements the catalog on DuckDB, uses gRPC with DuckDB’s internal serialization, and offers per-user instances that start in ~100 ms; largest nodes are 192 cores and 1.5TB RAM. Demo reproduces the Dremel 85 billion-row wordcount in about two minutes.

AI-written summary · about

" chapters (29)

1. 0:00Introduction
2. 1:25BigQuery to DuckDB and DuckLake
3. 3:10Cloud warehouse evolution
4. 6:41Lakehouse rise and limits
5. 8:25Lakehouse vs cloud warehouse
6. 9:49File pruning in BigQuery Iceberg DuckLake
7. 15:07ACID in BigQuery Iceberg DuckLake
8. 19:07Small writes handling
9. 20:13Lakehouse choices trade-offs
10. 23:00Design constraints metadata
11. 24:24Spark connector implementation
12. 25:49Demo performance scale
13. 26:11Managed DuckLake service
14. 27:57MotherDuck architecture tenancy
15. 31:10Managed DuckLake features access
16. 35:47Zen of data management
17. 36:09Q&A: Querying buffered small writes
18. 38:18Q&A: BigQuery catalog gaps
19. 39:19Q&A: Metadata caching performance
20. 40:24Q&A: Workloads transaction limits
21. 42:34Q&A: Scaling beyond DuckDB
22. 44:20Q&A: Storage optimization tiering
23. 46:05Q&A: Multi-user access routing
24. 47:33Q&A: Metadata storage openness
25. 48:38Q&A: Interoperability engines
26. 51:28Q&A: Access control model
27. 52:33Q&A: Column statistics generation
28. 53:37Q&A: MotherDuck optimizations
29. 54:20Q&A: Engineering priorities

" related talks

• 38mFrom Big Data to DuckDB: Ryan Boyd (MotherDuck) on Hyper-Tenancy, Duck Lake, and Making Data Fun.Ryan Boyd · AI By the Bay 2025
• 47mWhy DuckDB could be the engine for your Data Lake?Vincent Heuschling · Voxxed Days Luxembourg 2023
• 26mBuilding Data Lake Using DuckDBSubash Roul · DuckCon #4
• 51mMotherduck: Data Analytics in the Post Big-Data EraJordan Tigani · Compass Tech Summit 2023
• 28mAnalyze data easily with duckdb - and the implications on data architecturesMatthias Niehoff · PyCon DE & PyData 2025
• 22mData Exploration Made Easy with Open SourceOz Katz · Voxxed Days Luxembourg 2023
• 24mData infrastructure to build bigger with lessRyan Boyd · ODSC West 2024
• 06mA quack at building scalable data pipelines with DuckDBJunaid Rahim · DuckCon #5 (Seattle, 2024)