A Theoretical Approach to Semantic Coding and Hashing
summary
Proposes a log‑linear generative model where a latent discourse vector does a random walk on the sphere, words have vectors VW, and under spatial isotropy the partition function self‑normalizes so PMI ≈ inner product; this explains why 300‑dim embeddings fit a 100k×100k PMI matrix (≈17% entrywise error) and why Word2Vec/GloVe are MLE approximations. Shows relations align to directions, demonstrates polysemy mixes senses linearly so sparse coding can recover them, and gives a sentence embedding = weighted sum with weight A/(A+P_w) that outperforms prior unsupervised methods.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:09Semantic Hashing
- 6:14Word Embeddings
- 9:02Meaning
- 13:38History
- 19:25Why do word vectors exist
- 21:02Dynamic publication model
- 27:05Self normalization
- 27:46Lowdimensional vectors
- 29:46Embedding methods
- 30:15Weighted SVD
- 31:54Formalizing relation
- 33:38Polysemy
- 35:22Meaning Extraction
- 35:57Sentence Embedding
- 38:42Summary
related talks
16mMachine Learning on Source Code
36mThe unreasonable effectiveness of feature hashing
49mlda2vec
56mTensor Decompositions for Learning Latent Variable Models I
51mRandom Projections for Probabilistic Inference
50mWhat Comes After ChatGPT? Vector Databases - the Simple & Powerful Future of ML?
37mMin-Wise Hashing for Large-Scale Regression and Classification
13mA Beginner’s Guide to Semantic Search