ESPN: Memory-Efficient Multi-Vector Information Retrieval
summary
ESPN offloads multi-vector reranking embeddings from RAM to SSDs and uses GPU Direct Storage plus a software prefetcher to overlap compute and IO, reducing index memory by about 5–16× (authors report ≈8× in experiments). Prefetching 10% of candidate vectors yields a 90% hit rate, cutting SSD retrieval up to 6.4× and improving end-to-end throughput by 68%, with example query latency dropping from ~76 ms to ~46 ms after GDs, kernel and prefetch optimizations. Code and datasets are available on the project repo.
AI-written summary · about
chapters (10)
- 0:00Multi-vector retrieval intro
- 2:05How multi-vector IR works
- 4:11Memory challenges with inverted indexes
- 8:07Motivation for offloading index
- 10:14ESPN’s IO prefetching strategy
- 13:23Evaluating recall and hit rates
- 15:118x memory reduction with ESPN
- 16:39Performance results overview
- 17:41Limitations and future work
- 18:43Conclusion
related talks
28mThe Memory Efficient Indexing for Vector Databases
45mBehind the hype: managing billion-scale embeddings in Elasticsearch and OpenSearch
47mBuilding intelligent analytics at scale
01h 23mVector Databases Demystified: Discover how they work
33mTowards a deeper understanding of retrieval and vector databases
28mA Deep Dive into Python-Powered Precision and Scalability
25mVector Streaming: The Memory Efficient Indexing for Vector Databases
32mRetrieval Augmentation and Semantic Search at Scale