Bringing AI to DuckDB with Lance columnar format for multi-modal AI
summary
Lance columnar format and LanceDB aim to replace Parquet+TFRecords copies by storing embeddings, images, text, point clouds and nested annotations together with zero-copy schema evolution and versioning. Integrates with DuckDB, pandas, polars, PyTorch; adds FFmpeg/OpenCV scanners, CUDA-enabled UDFs and DuckDB extensions for model execution. Claims several orders of magnitude faster random access and 2–3x faster overall training from faster shuffling, filtering and data loading. Notes Arrow predicate-pushdown incompatibilities and missing AI types like annotations and extension binary types.
AI-written summary · about
chapters (5)
- 0:00Introduction and context
- 0:54Challenges in AI data
- 1:47Lance columnar format
- 2:41Building an AI stack
- 3:43Data types and challenges
related talks
28mBeyond Parquet and ORC: Upgrading Data Infrastructure for Multi-modal AI with Lance Col...
28mLanceDB: lightweight billion-scale vector search for multimodal AI
31mTransforming Multimodal Data Management with LanceDB-Ray
40mDuckDB: Harnessing in-process analytics for data science and beyond
28mAnalyze data easily with duckdb - and the implications on data architectures
31mAgents Need More than Vector Search
27mEmpowering PySpark with Lance Format for Multi-Modal AI Data Pipelines
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB