From Big Data to DuckDB: Ryan Boyd (MotherDuck) on Hyper-Tenancy, Duck Lake, and Making Data Fun.
summary
Argues that replacing massive distributed engines with many small DuckDB instances yields real wins: MotherDuck’s hyper-tenancy gives each customer or user a dedicated DuckDB, enabling thousands of simultaneous users without per-query sharding; dual execution runs identical DuckDB locally and in cloud so SQL previews appear in milliseconds and sub-second queries are routine. Proposes Duck Lake—store lakehouse metadata in Postgres to cut dozens of HTTP calls into a single 20–50 ms SQL lookup. Demonstrations cite a BigQuery job that needed 1,100 machines a decade ago now running on a laptop in ~50s.
AI-written summary · about
chapters (16)
- 0:00Introduction and excitement
- 1:51Why DuckDB is magical
- 3:38Hypertency and scaling with DuckDB
- 5:25Legacy architecture destruction
- 6:51Dual execution and local-first cloud
- 8:37Efficiency and resource constraints
- 10:44Innovation and academic collaboration
- 12:53MotherDuck's pre-AI wave strategy
- 15:45AI and text-to-SQL advancements
- 17:30Devrel and marketing strategy
- 21:45Open-source support and community
- 24:14Instance SQL and flow state
- 27:03Duckify and fun in data
- 29:11What is a data lake?
- 32:01Duck Lake simplification
- 37:39Closing and thanks
related talks
51mMotherduck: Data Analytics in the Post Big-Data Era
33mDataEngBytes 2023 - SYD-T1-01 - Ryan Boyd
55mDuckLake: Learning from Cloud Data Warehouses to Build a Robust “Lakehouse”
40mDuckDB Meets Data Lakes
36mMotherDuck
47mWhy DuckDB could be the engine for your Data Lake?
27mMotherDuck: Taking flight with interactive analytics
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB