Empowering PySpark with Lance Format for Multi-Modal AI Data Pipelines
summary
Claim: PySpark writes directly to Lance format using the Python Data Source API and Arrow so transformed data lands as Lance fragments without intermediate files. Built outcome: a distributed writer plus commit protocol that publishes manifests and works end to end. Performance notes: random access and index-backed filters score up to 1,000x faster than Parquet and the format supports millions of tables and trillions of rows, plus embedded Lens DB vector search and GPU streaming for images and video.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:29AI-driven data lakes
- 6:30Lance format for AI data lakes
- 10:05Lance as vector database
- 13:58Embedded and distributed LanceDB
- 16:51PySpark and Lance format
- 19:43Python Data Source API in Spark 4
- 21:07Lance PySpark connector
- 23:42Using the connector
- 24:48Q&A
related talks
31mTransforming Multimodal Data Management with LanceDB-Ray
28mBeyond Parquet and ORC: Upgrading Data Infrastructure for Multi-modal AI with Lance Col...
05mBringing AI to DuckDB with Lance columnar format for multi-modal AI
28mLanceDB: lightweight billion-scale vector search for multimodal AI
16mThe Post-Modern Data Stack for AI: Open Source Foundations for a Multimodal La...
34mLeveraging Apache Arrow for ML Workflows
24mPolyglot data with python: Introducing Pandas and Apache Arrow
46mA 101 in Time Series Analytics with Apache Arrow, Pandas and Parquet