Fuzzy Matching at Scale
summary
Claims that TF-IDF n-grams plus approximate nearest neighbors replace edit-distance methods for massive fuzzy matching, outperforming Levenshtein at scale. Implements a vector-space pipeline (char n-grams → sparse TF-IDF → ANN index) in a library called HotFuzz, running a 1,000,000-record index queried against 50k targets on a laptop in roughly coffee-break time. Index building is centralized, querying is the bottleneck, and scaling uses Spark for CPUs or RAPIDS for GPUs; results are practical and reproducible.
AI-written summary · about
chapters (9)
- 0:00Start
- 1:54String Operations
- 2:41Edit Distances
- 3:29Algorithm
- 7:08Implementations
- 8:20A Brief History of NLP-Vector Space Models
- 9:51Implementation of Fast Fuzzy Matching
- 11:53Cosine Similarity
- 13:20Next Steps
related talks
33mFuzzy Search Algorithms How and When to Use Them
24mSpeeding up search with locality sensitive hashing
38mA Bi-metric Framework for Fast Similarity Search
59mContextual search with vector search: exploring your options with open source tools
01h 09mVector Search: Ask Me Anything!
38mSearch: A New Era
28mFuzzyTM: a Python package for fuzzy topic models
49mSpectral Approaches to Nearest Neighbor Search