Advancing Diffusion Models for Text Generation
summary
Introduces Latent Diffusion for Language Generation and a pretraining recipe called LMLM that replaces tail facts with fake DB lookup tokens and forbids backprop on the returned values, trains the lookup annotations with GPT-4 over Wikipedia and processed all of Wikipedia in ~12 hours, then builds a key-value database and shows a 124M GPT-2 baseline 124M GPT-2 doubles fact score when trained this way, performance collapses if DB calls are disabled no backprop on DB returns, and latent diffusion plus semantic proposals yields STAR-style control that rivals models matching ~12x larger models.
AI-written summary · about
chapters (11)
- 0:00Large memory language models
- 5:32Data preparation pipeline
- 10:50Training and inference
- 13:37Evaluation and results
- 18:52Knowledge separation impacts
- 27:51Latent diffusion intro
- 33:45Latent diffusion for text
- 37:54Controllable generation
- 44:29Diffusion guided modeling
- 50:35Stop-think-autoregress
- 55:18Q&A and concluding remarks
related talks
28mText Diffusion
25mGenerative Text Modelling: Scratching the surface
53mGenerative AI: Overview and use cases
45mHow Generative AI Works: LLMs and RAG Under the Hood
52mUnleash the Power of Large Language Models (LLMs)
43mBeyond the Hype: A Realistic Look at Large Language Models
24mHarnessing the Power of LLMs in Advanced Analytics
31mEmpowering Open-Source Generative AI by Integrating the Wikidat...