The unreasonable effectiveness of feature hashing
summary
Feature hashing maps high‑cardinality categorical and bag‑of‑words features into a fixed, sparse vector with a cheap hash and modulo step, uses a sign bit trick to make collisions cancel in expectation, and benefits from power‑of‑two bucket sizes for ~30% faster indexing. Implementations include Vowpal Wabbit, scikit‑learn’s HashingVectorizer and Spark ML; a production use handles about 350 million observations with ~8.5 million coefficients, yielding a ~10% A/B lift and modest statistical degradation from collisions.
AI-written summary · about
chapters (18)
- 0:00Welcome!
- 1:00Introduction
- 1:53Background- Supervised ML
- 3:29Categorical Features
- 4:00One-hot encoding
- 4:36Bag of words
- 5:20High dimensional feature space
- 9:20Feature Hashing
- 11:26Hash function
- 12:24Feature Hashing in Python
- 13:27Hashing of Unicode Strings
- 15:06Collisions
- 17:58Sign Functions
- 20:26Feature Hashing- Example
- 26:40Feature Hashing- Use Case
- 30:10Library Support
- 31:27Recap
- 32:20Q&A
related talks
39mProbabilistic Hashing Methods for Succinct Data Representations and Applications
37mMin-Wise Hashing for Large-Scale Regression and Classification
32mAdventures with Randomized Algebra for Extreme-Scale Signal Processing
01h 02mImportance Sampling in High Dimensions via Hashing
42mWhen Hashes Met Wedges - A Distributed Algorithm for Finding High Similarity Vectors
11mFinding Friends in High Dimensions...
26mUsing Random Projections to Make Sense of High-Dimensional ...
44mA Theoretical Approach to Semantic Coding and Hashing