Analyze data easily with duckdb - and the implications on data architectures

on YouTube ↗28m

" summary

DuckDB is an in-process analytical engine that queries Parquet/JSON over HTTP and S3 (example: reads a 2.5 GB JSON blob) and can attach tiny catalog files (few KB) that point to distributed sources. It interoperates with Python via Pandas/Polars/Apache Arrow with near-zero conversion cost and runs in the browser via WASM execution. Teams replace slow PySpark unit tests and some ETL jobs (claims of hours→minutes) and run DuckDB in Lambda; limitations remain: single-node, limited multi-user and write support for some lakehouse formats. Version 1.3 targets Iceberg/Delta performance.

AI-written summary · about

" chapters (11)

1. 0:00Introduction to DuckDB
2. 1:02Demonstrating core features
3. 7:03Managing persistent databases
4. 8:33Python and ecosystem integration
5. 11:29Web and browser capabilities
6. 12:52Conceptual architecture
7. 14:15Flexibility and extensions
8. 17:00Practical use cases
9. 18:56Benefits and limitations
10. 22:15Impact on data architecture
11. 24:16Conclusion and Q&A

" related talks

• 27mIn Process Analytical Data Management with DuckDBAlex Monahan · Scale By the Bay 2023
• 34mData Wrangling with DuckDBWill Angel · PyData Virginia 2025
• 01h 02mAnalytics for not-so-big data with DuckDBDavid Ostrovsky · NDC Oslo 2025
• 40mDuckDB: Harnessing in-process analytics for data science and beyondGabor Szarnyas · Øredev 2023
• 40mDuckDB beyond the notebookMatthias Niehoff · Berlin Buzzwords 2026
• 27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDBAnkur Jain · PyCon AU 2025
• 25mA quack of all trades: DuckDB, a versatile analytical database to keep in your toolkitSimon Aubury, Ned Letcher · DataEngBytes Melbourne 2024
• 19mIn-Process Analytical Data Management with DuckDBHannes Mühleisen · posit::conf(2023)