Text and Code Embeddings
summary
Explains that OpenAI trains general-purpose text and code embeddings by contrastive training with very large batch sizes, initializing models from GPT/Codex and using other batch examples as negatives, which yields scalable indices for up to 10 billion items via a vector database. Reports empirical wins: strong linear-probe results on SentEval-style tasks and BEIR-style retrieval benchmarks and a cited 20% relative improvement on code search. Describes retrieval-augmented generation with ChatGPT plugins, LangChain, and open-source cookbooks.
AI-written summary · about
chapters (8)
- 0:00Embeddings
- 2:01Use Cases
- 4:17Brief history of embeddings in the last decade
- 6:56Contrastive Training
- 13:38Recipe
- 15:33Experiments
- 21:17Retrieval
- 26:10Future Directions
related talks
16mEmpower AI: Building Smarter Solutions with Embeddings
03mContextual understanding using Generative Artificial Intelligence
45mScalable Open AI GPT Applications in the Cloud
45mBringing the Power of AI to Your Application
48mUnderstanding Embedding Vectors and the RAG Pattern in AI Solutions
25mEmbeddings, Transformers, RLHF: Three key ideas to understand ChatGPT
51mOpenAI and the AI that may overtake humans
32mGenerative AI Text and Multimodal Embedding Models for Real World Use Cases