Implementing Lightweight Random Indexing for Polylingual Text Classification -PyData NYC
summary
Implements lightweight random indexing to merge multilingual TF‑IDF corpora into one training matrix by assigning each token a sparse random index vector with two nonzero entries drawn from ±1/√2, accumulating vectors per document instead of doing a large projection multiply. Vocabularies are aligned into shared and language‑specific blocks, producing a single classifier that preserves relative distances. Experiments on ~8,000 Reuters stories across five languages with 15,000 common terms (≈100,000 features) outperform a machine‑translation baseline.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:28Text classification
- 3:39Multilingual text challenges
- 5:27Random projections overview
- 9:28Random indexing in practice
- 12:39Lightweight random indexing
- 15:50Multilingual data application
- 17:34Experimental setup
- 20:22Results and comparison
- 22:09Q&A
related talks
48mIntroduction to Natural Language Processing with Python
30mKick-Start your Understanding of Machine Learning with Python
31mTaking AI/ML to the next level: Text!
31mKeynote Q&A – Barbara Plank
30mFuzzy Matching at Scale
16mDense Vectors in Elasticsearch
26mUsing Random Projections to Make Sense of High-Dimensional ...
01h 01mWord Embeddings - An Alternative & Efficient Approach to Search for Docs