Semantic Label Representation and Multimodal Categorization
summary
Product categorization uses CLIP embeddings to get category name vectors, computes cosine similarities, and truncates to top-20 nearest neighbors to build soft targets rather than one‑hot labels. Training mixes focal cross‑entropy with a KLD auxiliary loss toward those soft distributions. Four label schemes are compared; the semantic nearest‑neighbor smoothing boosts top‑1/top‑5 accuracy and raises the fraction of examples whose all top‑5 predictions lie inside the top‑20 neighbors from about 1.55% to 57.1%.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:19Classifier goals in practice
- 5:11Multimodal product categorization
- 8:22Baseline model setup
- 9:05One-hot encoding limits
- 11:58Label smoothing for overfitting
- 13:02Semantic label smoothing
- 16:38Semantic distances with CLIP
- 18:04Experimental results
- 22:19Key takeaways
- 24:24Loss function intuition
- 25:49Q&A
related talks
33mBulk Labelling Techniques
26mMaximizing Multimodal: Exploring the search frontier of text-to-image models
38mE-Commerce Product Categorization at Scale
41mSemantic Multimodal Multilingual Similarity Engine
20mPragmatic Deep Learning for image labelling
56mApplications of Multimodal Learning in Media Search Engines
39mMaking Use of All Your Data, Labelled or Otherwise
38mLeverage Reviews Data for Multi Label Topics Classification