A quack at building scalable data pipelines with DuckDB
summary
Atlan describes a Kubernetes-native pipeline using Argo Workflows to run embedded DuckDB instances that read and write Parquet artifacts on S3. A TPC-like dataset is ~100 GB in memory and compresses to ~39 GB Parquet, split into ~3.3 GB chunks handed to per-pod queries; DuckDB runs about twice as fast as the prior PySpark setup under these limits. The system applies predicate pushdown, memory/thread limits, dynamic pod sizing and recursive DAGs and it succeeds at lean, scalable analytics.
AI-written summary · about
chapters (9)
- 0:00Introduction to Argo
- 0:26Argo workflow artifacts
- 1:21Data pipelines at Atlan
- 1:48Architecture and data
- 2:38TPC-DS benchmark example
- 3:02Performance results
- 3:36Pipeline benefits
- 4:27Dashboarding and summary
- 5:05Q&A session
related talks
26mBuilding Data Lake Using DuckDB
28mAnalyze data easily with duckdb - and the implications on data architectures
45mAnalytics for not so big data with DuckDB
25mA quack of all trades: DuckDB, a versatile analytical database to keep in your toolkit
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers
51mMotherduck: Data Analytics in the Post Big-Data Era
05mDump the Ducktape: Unblocking data pipelines at scale in healthcare using
30mSparkless Patterns: The New Data Stack and Modern Data Architecture