Entity Linking at scale with Lucene
summary
Signal AI replaces regex-heavy rule matching with Apache Lucene using a percolator model and custom analyzer so documents are tokenized once and queried many times. The system moves from ~10k rules and 2M docs/day with p99 ≈10s and $2k/month to ~110k rules, 5M+ docs/day, p99 ≈8s for 50-doc batches (~200ms/doc) and monthly cost ≈$200. Nightly reprocessing uses 50k batches and prebuilt indexes to finish in ~15 minutes, following compute once consume many.
AI-written summary · about
chapters (15)
- 0:00Introduction and overview
- 2:10Entity extraction at Signal AI
- 6:43Scaling challenges in 2019
- 8:49Performance bottlenecks in regex
- 11:59Why Apache Lucene
- 13:47Implementing Lucene solution
- 16:38Results after Lucene adoption
- 17:27Compute once consume many paradigm
- 20:58Scaling with three principles
- 24:28Real-time processing at scale
- 26:53Offline processing challenges
- 28:16Architecture for offline processing
- 31:08Offline reprocessing improvements
- 32:31Key takeaways and summary
- 32:53Q&A session
related talks
25mApache Lucene: From Text Indexing to Artificial Intelligence
37mAlgorithms & data-structures that power Lucene & ElasticSearch
20mMultilingual Search System - How to Build and Improve!?
42mBuilding a real-time news search engine #bbuzz
45mSafely Capture Institutional Knowledge Using
19mVectorize Your Open Source Search Engine
36mSearch on the fly: how to lighten your Big Data
12mText categorization with Apache Lucene