Things I Learned the Hard Way Building a Search Engine
summary
Replaced a PostgreSQL text search with Elasticsearch and reconstructs why ranking needs more than keyword matching: builds an inverted index, vectorizes documents and queries, applies cosine similarity with TF‑IDF + pivot length normalization, adds stop‑word removal, stemming/lemmatization, synonym lists and query‑expansion, boosts exact‑term and full‑term‑coverage matches, and considers misspellings. Demonstrates the model on a toy corpus, shows better ordering but some surprising mismatches, and notes BM25f as an alternative for record‑like data.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:10PostgreSQL to Elasticsearch
- 2:39What is information retrieval
- 5:36How information retrieval works
- 9:55Building an inverted index
- 11:49Text similarity with vectors
- 15:07Stop words and stemming
- 17:18Term frequency in ranking
- 20:20Document frequency explained
- 21:48TF-IDF for relevance
- 23:11Normalizing document length
- 24:58Final ranking function
- 26:05Synonyms for better results
- 27:56Ranking thresholds
- 29:19Text search principles
- 31:26User search intent
- 33:14Q&A
related talks
40mSmarter Search with Spark, Solr and Machine Learning
47mChoosing a Search Engine with Your Head
37mBuilding advanced RAGs with Elasticsearch OpenSearch
01h 01mWord Embeddings - An Alternative & Efficient Approach to Search for Docs
20mLayering every technique in RAG, one query at a time
39mCross Domain Enterprise Search - Content Diversity at Scale
01h 22mBoot your search with Spring
01h 02mWhy It Won't Be Found!