Applying contrastive learning to fight anisotropy in language models
summary
Claim: applying SimCSE (contrastive objective) with dropout augmentation turns cone-like BERT/Roberta embeddings into a more spherical, isotropic distribution. Tools: Transformer encoders and the Python SimCSE library; implementation simply calls encode and compares cosine similarities. Result: paraphrase pairs show substantially higher cosine scores versus unrelated pairs (example pair ~0.14 for unrelated vs noticeably higher for paraphrases), and supervised SimCSE empirically balances alignment and uniformity better than whitening.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 2:14Transformers and BERT
- 4:00Agenda
- 5:06Transformer refresher
- 8:57Polysemy in language
- 10:41Word embeddings
- 12:27Isotropy definition
- 14:55Anisotropy in embeddings
- 16:45Consequences of anisotropy
- 17:06Contrastive learning intro
- 19:09Contrastive learning methods
- 20:12Unsupervised sentence learning
- 21:40Embedding alignment
- 23:04Evaluating isotropy
- 24:08Python SimCSE
- 25:10Applications and demos
related talks
25mSemantic Search
28m"Embeddings: How Computers Learned to Read"
29mHallucination by Design: How Embedding Models Misunderstand Language
30mBiases in Language Models
41mSemantic Multimodal Multilingual Similarity Engine
01h 28mDocument/Sentence Similarity Solution
52mLanguage-guided Adaptation
39mAI-powered Semantic Search; A story of broken promises?