Transforming Multimodal Data Management with LanceDB-Ray
summary
LanceDB presents a columnar, object-store native format that stores images, video and embeddings inline and adds columns without rewriting data, enabling zero-copy schema evolution and large blob support. It builds secondary and vector indexes for fast random access and reports millions of ops per second on clustered setups; indexing is decoupled from writes and compaction produces new immutable versions so old snapshots remain available. Native Ray integration streams filtered, shuffled subsets from S3/GCS for distributed training and large-scale indexing.
AI-written summary · about
chapters (7)
- 0:00Introduction and welcome
- 3:00Why AI needs new foundations
- 8:22Introducing LanceDB
- 10:11LanceDB’s architecture
- 14:26Vision for unified data workflow
- 16:16LanceDB-Ray integration
- 19:29Q&A session
related talks
28mLanceDB: lightweight billion-scale vector search for multimodal AI
31mAgents Need More than Vector Search
28mBeyond Parquet and ORC: Upgrading Data Infrastructure for Multi-modal AI with Lance Col...
05mBringing AI to DuckDB with Lance columnar format for multi-modal AI
16mThe Post-Modern Data Stack for AI: Open Source Foundations for a Multimodal La...
27mEmpowering PySpark with Lance Format for Multi-Modal AI Data Pipelines
17mScaling Enterprise-Grade RAG: Lessons from Legal Frontier
26mData Processing on Ray