Near-Duplicate Ad Detection in Online Classified Ad Services
summary
Problem definition, datasets and concrete gains: models train on ~1M labeled pairs with 200–300 test pairs and production sees ~500k submissions/day and 60% image-hash coverage. Text baselines (CountVectorizer) evolve to Bi-LSTM with attention and adding embedding differences yields ~+8% ROC AUC (from 0.83 to ~0.93) then heavy feature engineering (Levenshtein, MinHash, common numbers) adds a few points and image signals via pHash and MobileNetV3 + Milvus add ~2% bringing final ROC AUC to ~0.959. Pipelines include elastic services for near-duplicate retrieval and a pairwise classifier constrained by operational recall targets.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 2:00Global vs local duplicates
- 4:08Local duplicates and intentions
- 8:05Problem formulation
- 9:54Base model architecture
- 13:11Text-only limitations
- 15:41Version 2 text features
- 17:30Improving embeddings
- 18:37Image features with P-hash
- 21:07Feature engineering
- 23:14Final model results
- 24:18Future directions
- 25:46Q&A
related talks
33mMachine Learning to moderate ads
36mFinding needles in haystacks with Deep Neural Networks
22mHarnessing image and text data for product matching
46mGarbage data in, garbage models out
01h 15mTrain Object Detection with Small Datasets
19mMachine Learning for Privacy-Preserving Ads
28mVisual Search
34mComputer Vision and NLP for Multi-Task Fashion Attrb. Modeling