From Naive Sorting to Natural Language Processing, 3 Years of Optimization of Our Search Engine
summary
The talk explains how a job-search engine evolved its ranking from simple date-only sorting to market-specific configurable scoring using Elasticsearch primitives (TF‑IDF, time decay), Kafka ingestion, and per-market indices. It then describes adding neural techniques: embedding search with CamemBERT/BERT plus ANN/KNN to address synonyms, which improved contact rates and reduced zero-result queries but introduced 5s latencies and higher disk/RAM and reindexing costs on AWS.
AI-written summary · about
chapters (11)
- 0:00Project introduction
- 2:28Initial engine problems
- 6:23Platform building
- 13:44First optimizations
- 19:15Semantic search and ML
- 27:31Precision improvement
- 33:34NN search integration
- 37:05NN search results
- 40:16Infrastructure optimization
- 42:05Advice and conclusion
- 44:32Q&A
related talks
01h 48mInformation Retrieval from the Ground Up
42mResearch in the Age of AI
30mSearch on Steroids — A Story of Semantics
25mApache Lucene: From Text Indexing to Artificial Intelligence
45mIntegrate your Search Engine (...)
15mHow Big Data Makes Content More Relevant in Search
19mBuilding a Smarter AI Agent with Neural RAG
19mYou Know, for Search. Querying 24 Billion Records in 900ms