State of the Duck
summary
Progress report on storage, execution, and adoption: the project ships persistent ART indexes, parallel bulk loading and many new compression codecs that shrink datasets toward Parquet sizes, and implements out-of-core hash join plus out-of-core sorting and window operators so large queries stream to disk instead of OOMing. Releases add attach/multi-database and a SQLite storage extension, parallel JSON and Hive-partitioned read/write, delta compression, upserts and lateral/positional joins. Adoption jumps to 1,000,000 Python installs/month and JDBC downloads rise from ~6k to ~600k.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:03Goals recap
- 8:08Storage improvements
- 14:07Memory management
- 18:22Performance optimizations
- 21:48New features
- 24:39Upcoming features
- 33:24Growth and adoption
- 39:03Foundation and governance
- 40:47Roadmap
- 44:20Q&A
related talks
22mData Exploration Made Easy with Open Source
05mAnnouncing DuckDB Support for Delta Lake and a DuckDB Extension to Unity Catalog
35mTen years of building open source standards.
15mWelcome and opening remarks
22mDat
28mOpen Source meets Enterprise: The right way.
27mDave Stokes talks open - Stocking the flames of open source - Episode 1
25mState of Open Source Databases