Apache Lucene: From Text Indexing to Artificial Intelligence
summary
Apache Lucene is presented as a long-maintained search library that powers Elasticsearch, OpenSearch, Solr and MongoDB Atlas Search and is widely deployed (LinkedIn, Twitter). The talk describes Lucene’s core technologies—inverted index with tokenization and analyzers, doc values (a column store) for fast sorting and faceting, geospatial indexes—and the newer dense vector search for several-hundred-dimension vectors. The presenter reports up to 100× performance gains on some features and argues Lucene can serve as the vector database in RAG pipelines to reduce LLM hallucinations.
AI-written summary · about
chapters (9)
- 0:00Introduction to Apache Lucene
- 3:15Lucene history and community
- 7:30Inverted index operation
- 10:23Text analysis and autocompletion
- 13:15Column storage and faceting
- 15:02Geolocated search
- 16:26AI and vector search
- 20:21RAG with Lucene
- 23:13Conclusion and resources
related talks
01h 04mApache Lucene 8, Apache Solr and Elasticsearch – an Overview
37mAlgorithms & data-structures that power Lucene & ElasticSearch
45mAnything New in Search? What If I Find It?
46mFull-Text Search Internals with Elasticsearch
12mText categorization with Apache Lucene
01h 00m50 Shades of Join in Apache Lucene
36mSearch on the fly: how to lighten your Big Data
55mFull-Text Search Demystified