The Duck and the DataFrame: A Data Engineer’s Journey with DuckDB
summary
Claims that DuckDB replaces heavy distributed tooling for moderate data sizes and shows concrete results: a dbt pipeline reads Hive‑partitioned Parquet on S3, runs DuckDB as the execution engine, and builds bronze/silver/gold models with 3 months = 61M records / 36 GB compressed / 142 GB uncompressed. Benchmarking on an EC2 M6a/M62x‑class node and a local run with 14 GB RAM completes seven models in ~5m22s. Notes limits: single‑node, not for write‑heavy streaming, and requires custom macros to read/write S3.
AI-written summary · about
chapters (12)
- 0:00Welcome
- 1:45Why ducks are cool
- 2:50State of data pipelines
- 4:57Distributed computing overkill
- 6:00What is DuckDB
- 8:45DuckDB vs Spark
- 9:49DuckDB with dbt
- 10:54Live demo: GitHub data
- 18:46Limitations of DuckDB
- 19:49DuckDB extensions
- 20:30Key takeaways
- 21:34Q&A
related talks
28mAnalyze data easily with duckdb - and the implications on data architectures
01h 02mAnalytics for not-so-big data with DuckDB
40mDuckDB beyond the notebook
34mData Wrangling with DuckDB
40mDuckDB Meets Data Lakes
51mMotherduck: Data Analytics in the Post Big-Data Era
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
26mBuilding Data Lake Using DuckDB