Tune-Up Your Knowledge: A Beginner's Journey Under the Hood of Search Engines
summary
Builds a search engine in Python over ~6,200 Amazon book records, showing how naive tokenization returns 16 matches for "Python" then improves to 410 after case folding and 537 after punctuation stripping, then 314 when queries aren’t normalized; constructs an inverted index with ~87,000 unique terms to avoid full scans; implements TF‑IDF and BM25 to rank results and demo differences; adds 384‑dim vector embeddings and cosine similarity for semantic matches, noting vectors always return nearest neighbors and model choice drives quality.
AI-written summary · about
chapters (15)
- 0:00Search engine internals
- 1:03Personal journey into search
- 3:08Lexical search pillars
- 5:55Tokenization matters
- 8:48Matching query tokens
- 10:36Inverted indexes speed search
- 13:28AND/OR logic in search
- 15:15TF-IDF scoring
- 17:01Ranking with TF-IDF
- 18:03BM25 relevance upgrades
- 20:38Lexical search limits
- 21:41Semantic search with vectors
- 24:10Vector search and cosine
- 25:55Lexical vs semantic search
- 26:57Key takeaways and resources
related talks
24mSearch From the Ground Up
05mHow to build a Code Search Engine
01h 39mNatural Language Processing (NLP) in Python - From Zero to Hero
30mKick-Start your Understanding of Machine Learning with Python
26mFrom __icontains to search
24mHarry Potter and the Elastic Python Clients
26mMeasuring Search Engine Quality using Spark and Python
27mWoosh: a fast pure-Python search engine library