Towards fast and coherent long-form text generation
summary
Claims that replacing learned self-attention with hard-coded attention yields similar quality while cutting decode time substantially; experiments use GPT-2 and transformer variants and report a >30x throughput gain in some settings and 10s baseline for 300→300 tokens on a 2080Ti. A story-suggestion system on the Storium platform serves ~30k users with 4–8 GPUs; users keep short phrases and delete suggestions half the time. Model extraction attacks recover BERT/GPT-quality models for about $62.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 4:23Story generation
- 8:18Story generation model
- 14:03Language models
- 16:28User interaction
- 19:01Challenges
- 20:33Timing comparison
- 22:47Selfattention
- 26:17Hardcoded selfattention
- 28:46Transformer internals
- 31:09Efficiency
- 33:16Security
- 35:22Presentation
- 35:34Sentiment analysis
- 36:42Stolen model
- 39:13Financial practicality
- 39:59Random queries
- 41:51Results
- 43:09Conclusion
- 44:32Thank you
related talks
25mGenerative Text Modelling: Scratching the surface
28mGPT or how I learned to stop worrying and love the generative AI
45mTips For Detecting The Use Of AI-Generated Text
01h 00mInside GPT – Large Language Models Demystified
45mHow to use GPT in your projects: A Case Study
52mUnleash the Power of Large Language Models (LLMs)
18mHow to Make Small Language Models Work
49mBen Hall & Barbara Fusinska