Use dbt and DuckDB Instead of Databricks for Data Processing
summary
Claim: dbt + DuckDB replaces Databricks for many medium workloads by running DBT SQL run dbt in-process against Parquet/Iceberg on blob storage; benchmarks show DuckDB outperforms Spark up to 100 GB and handles out‑of‑core work. Implementation: containerised dbt images scheduled as Azure Container Apps jobs, reads from Azure Blob, writes Parquet/Iceberg, and interoperates with Unity Catalog. Production wins on cost and local experimentation; losses are single‑node scalability beyond ~100 GB and more platform wiring versus an all‑in‑one Databricks setup.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:48Architecture over vendor choice
- 5:00Evaluating Databricks
- 13:05Introducing DuckDB
- 19:48Hybrid architecture
- 23:00Demo comparing stacks
- 32:32Summary and trade-offs
- 35:20Q&A library management
- 41:15Q&A output formats
related talks
28mAnalyze data easily with duckdb - and the implications on data architectures
22mDuckDBT: Not a database or a dbt adapter but a secret third thing
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB
26mBuilding Data Lake Using DuckDB
45mAnalytics for not so big data with DuckDB
24mpg_duckdb in Action: Accelerating Analytics on Azure Database for PostgreSQL
25mProcessing Biggish Data with DuckDB and Python
40mDuckDB beyond the notebook