Building an AI/ML powered text search system
summary
Takes 294 Berlin Buzzwords talks, extracts text with Apache Tika, turns it into tf-idf vectors (≈30,000 distinct terms), trains a multinomial Naive Bayes classifier and runs k‑means repeatedly to pick ~50 clusters, uses t‑SNE/PCA for visualization and then reduces to dense inputs via word2vec embeddings for neural nets. Demonstrates an IPython notebook, shows simple recommender effectiveness, warns about memory and scaling (Lucene/ElasticSearch wins at huge scale) and lists hyperparameters to record.
AI-written summary · about
chapters (18)
- 0:00Introduction and context
- 1:11AI and ML buzzwords explained
- 5:05Text data challenges
- 9:14Text preprocessing for ML
- 12:25ML basics for text search
- 16:25Building text similarity model
- 19:55Live demo of search system
- 21:02Clustering for text grouping
- 28:59Improving clustering results
- 30:51Incorporating time factors
- 33:23Tokenization and stop words
- 36:13Feature extraction and embeddings
- 38:44Neural networks for text
- 41:09Model training and evaluation
- 43:15Error handling and bias
- 45:43Natural language processing
- 46:27Comparison with Lucene Elastic
- 47:11Resources and Q&A
related talks
01h 14mBuilding a Recommendation System with a Hybrid Search From Scratch
21mBuilding RAG With OpenSearch ML Plugins: From PDFs To Voice-Enabled Search
41mAI is here – time to throw away our search engines?
34mLearning about AI/ML for Text, with Wordle!
43mAI enhanced UX
52mAI for Java Developers - From Buzzword to Code
55mSupercharged Search with Semantic Search and Vector Embeddings
02h 51mBuilding Your Intelligent Assistant with Hugging Face and Elasticsearch