Supercharging Multimodal Feature Engineering
summary
Claim: Lance plus Ray actually lets pipelines add new features by writing new column files not by rewriting whole tables and run UDFs across fragments with Ray workers. It demonstrates data evolution via an add_columns API that creates new fragment files (compaction later removes small‑file waste), a blob encoding that stores images inline with block APIs for GPU training, distributed index building and compaction on Ray, and ongoing work to slice fragments for better load balance and vector index building.
AI-written summary · about
chapters (15)
- 0:00Multimodal feature engineering intro
- 1:29What is feature engineering
- 3:57Traditional feature engineering challenges
- 6:25Parquet and Iceberg limitations
- 10:37Lance for multimodal storage
- 13:30Data evolution in Lance
- 17:45Blob encoding and multimodal data
- 18:08Next-gen multimodal lakehouse
- 20:13Scaling compute with Ray
- 23:23Why Ray for feature engineering
- 24:45Lance Ray integration
- 26:50Adding features with Lance
- 28:16Enterprise governance in Lance Ray
- 31:23Geneva advanced features
- 32:50Q&A
related talks
28mBeyond Parquet and ORC: Upgrading Data Infrastructure for Multi-modal AI with Lance Col...
05mBringing AI to DuckDB with Lance columnar format for multi-modal AI
33mHow Exa Indexes the Entire Web: Ray Data and Lance at Billion-Document Scale
30mSimplify Big Data & AI on Spark and Ray with MLSQL
28mLanceDB: lightweight billion-scale vector search for multimodal AI
34mLeveraging Apache Arrow for ML Workflows
33mEvolution and Scaling of Feature Store at Uber- 33mEvolution and Scaling of Feature Store at Uber