Understanding (a bit about) hallucinations in Generative AI
summary
Diffusion models trained on a simple shapes dataset and on hands produce 1–2% clearly impossible samples (two shapes in one region, nonfive fingers). Analysis shows neural nets smooth discontinuous true score functions, creating mode interpolation between nearby modes. A practical detector — the variance of the predicted x0 across the sampling trajectory — separates hallucinations from valid samples better than likelihood, and filtering by this metric reduces iterative data pollution and improves FID on MNIST.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 3:06Hallucinations in generative AI matter
- 13:30Diffusion model hallucinations
- 39:27Mode interpolation in diffusion models
- 55:52Evaluating hallucination detection
- 1:09:44Implications for model training
- 1:11:55Closing remarks
