How Exa Indexes the Entire Web: Ray Data and Lance at Billion-Document Scale
summary
Exa builds a petabyte-scale derived-data engine that stores documents in Lance’s two-dimensional disk layout so columns grow independently and new column files commit without rewriting existing data. The planner diffs Lance fragments to drive fragment-level patching and avoid write amplification, then compiles DAGs to Ray where Ray Actors enable stateful model reuse for pipelined CPU/GPU work. Concrete results: indexing a few billion rows in ~14 minutes on under 10 nodes and serving about 30,000 QPS from a single installation.
AI-written summary · about
chapters (12)
- 0:00Introduction to Exa
- 1:54Data processing challenges
- 5:06Logical layer with XAD
- 9:38Storage layer with Lance
- 15:02Execution layer with Ray
- 20:42Q&A on search quality
- 22:50Competing with Google
- 24:46Indexing multiple versions
- 26:34Performance metrics
- 27:16Building your own search
- 29:23GPU pipeline optimization
- 31:53Further resources
related talks
31mTransforming Multimodal Data Management with LanceDB-Ray
17mScaling Enterprise-Grade RAG: Lessons from Legal Frontier
42mSupercharging Multimodal Feature Engineering
35mHow Bytedance Builds Large-Scale Data Processing Pipelines for Multimodal Models with Ray
26mData Processing on Ray
28mBeyond Parquet and ORC: Upgrading Data Infrastructure for Multi-modal AI with Lance Col...
28mLanceDB: lightweight billion-scale vector search for multimodal AI
16mWhy we had to Build a Petabyte Data Lake to Fix Indexing