Hallucination by Design: How Embedding Models Misunderstand Language
summary
Claims that off‑the‑shelf embeddings misread negation and numerics and that experimentation fixes it; shows 38 concrete failure cases and examples where MPNet / MS Marco / sentence-transformers yield cosine similarity 96% → 46% for negation after fine‑tuning. Describes preprocessing (expand dates and numbers, add metadata), domain‑aligned models, and a fine‑tuning loop with training/validation/test, loss and evaluator, and reports lower false similarities and some negative returns. Tools shown: Hugging Face models, OpenAI examples, sentence-transformers code and evaluation.
AI-written summary · about
chapters (12)
- 0:00Introduction to Embeddings
- 0:16Understanding Embeddings
- 1:20Evaluating Embeddings
- 1:48The Embedding Process
- 3:53Key Concepts in Embeddings
- 6:27Use Cases for Embeddings
- 7:51Comparing Embeddings
- 8:42Practical Examples with OpenAI
- 13:27Challenges and Hallucinations in Embeddings
- 18:04Fine-Tuning Embedding Models
- 24:55Steps for Fine-Tuning
- 27:17Key Takeaways and Conclusion
related talks
34mTerrible tokenizer troubles in large language models
18mThe Unsung Hero of Vector Database: Metric Learning
45mVenturing out into the world with AI
48mKeynote – Sameer Singh
25mTricking Neural Networks : Explore Adversarial Attacks
54mKeynote: Can you trust your (large language) model?
01h 04mCan you trust your (large language) model?
41mTruth Checker: Generative Large Language Models and Hallucinations