The Duck(DB) Feather in Your Parquet Cap
summary
Company uses DuckDB in production since 0.41 to inspect Parquet files via duckdb parquet_schema and parquet_metadata, generate manifests, and emit SQL-based transforms that convert problematic types (notably decimals stored as byte arrays) into destination-compatible representations. Planner runs as an embedded library inside Kubernetes workers (no centralized service), produces transfer plans and executes Parquet-to-Parquet rewrites on S3, avoiding API/JSON row-by-row movement and achieving hundreds of megabytes per second when source and destination are Parquet. System is written mostly in Go and prioritizes planner library embedded execution for security and testability.
AI-written summary · about
chapters (11)
- 0:00DuckDB introduction
- 1:04Enterprise data sharing
- 3:29How Prequel solves it
- 5:14Data movement planning
- 7:00Type integrity in transfer
- 9:28Parquet for data movement
- 10:54Parquet type pitfalls
- 12:40DuckDB metadata queries
- 17:31DuckDB for type transforms
- 19:58Embedded DuckDB benefits
- 20:18Q&A
related talks
28mAnalyze data easily with duckdb - and the implications on data architectures
01h 02mAnalytics for not-so-big data with DuckDB
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB
40mDuckDB beyond the notebook
40mDuckDB: Harnessing in-process analytics for data science and beyond
19mIn-Process Analytical Data Management with DuckDB
24mpg_duckdb: Ducking awesome analytics in Postgres
25mA quack of all trades: DuckDB, a versatile analytical database to keep in your toolkit