Latest Techniques of Entity Matching in NLP
summary
Explains that end-to-end entity matching uses blocking to cut pairwise comparisons dramatically — e.g., a 10-record example drops from 45 expensive comparisons to 5 when k=2 blocking cuts 45 comparisons to 5 in a 10-record example — and presents a scalable deep-learning blocker: auto-block with FastText and attention-based tuple signatures trained to cluster matching records by maximizing cosine similarity, with ANN search (n≈15) for candidates. Production workflow uses entity-embed (pip install entity-embed) built on PyTorch Lightning requiring cluster IDs, returns embeddings and nearest-neighbor pairs, and pairs with LightGBM or human-in-the-loop thresholds (50–80%) and business metrics (e.g., 50% fewer duplicate pre-orders).
AI-written summary · about
chapters (4)
- 0:00Start
- 1:31What is entity matching?
- 10:38Pattern Matching/Fuzzy Matching
- 20:35How to measure success of Entity Matching?
related talks
38mA Bi-metric Framework for Fast Similarity Search
22mHarnessing image and text data for product matching
22mMachine Learning: The Engine Behind Modern E-commerce Search
49mNamed-Entity Recognition and Name Resolution Using Similarity Queries
30mFuzzy Matching at Scale
45mTransformer based clustering: Identifying product clusters f...
35mBuilding Named Entity Recognition Models Efficiently Using NERDS
39mHerding Entities- Information Search and Synthesis in the Context of Transaction Data