Repetita Non Iuvant: Why Generative AI Models Cannot Feed Themselves
summary
Claim: feeding models their own outputs progressively stable diffusion trained with 70% real 25% generated collapses after five retrains and OPT shows degeneration after nine self-training iterations; tools: Stable Diffusion, variational autoencoders, OPT, watermarkers like SynthID/VideoSeal; numbers: VAE shows mode collapse after ~20 generations, Satya Nadella’s cited ~30% AI‑written code underscores scale; fix tested: accumulation vs substitution dataset strategies reduce error, and Google’s escaping collapse protocol (K re-labels + beta weak labeler) provably converges to an optimal LLM.
AI-written summary · about
chapters (12)
- 0:00Generative AI pitfalls
- 1:53Iterative training trap
- 5:14Model collapse explained
- 8:14Finite sampling effects
- 10:45Visual evidence of degeneration
- 13:00Fixing with retraining or watermarking
- 15:49Accumulation vs substitution
- 18:43Escaping collapse with filtering
- 23:29Conclusion and future
- 24:14Q&A on AI code/data
- 25:58Q&A on cross-model contamination
- 27:58Q&A on data diversity
related talks
41mBuilding Generative Image & Video models at Scale
46mCan Humans’ Verification of Synthetic Data Save Models from Collapse?
26mGenerative AI: Get a Grip … and a Look Under the Hood
22mGenerative AI & its Transformation of Business, Research, and Society
27mChallenges of Generative AI
27mGenerative AI: What's Next?
44mKeynote: Tracking the Pulse of Generative AI
01h 21mGenerative AI with Diffusion Models