Talk by Mahdi Soltanolkotabi (University of Southern California)
summary
Adds trainable input tokens to a frozen foundation model (prompt-tuning) and analyzes a single‑head attention proxy on a Gaussian mixture style data model. Constructs distributions where prompt‑attention attains 100% classification while full self‑attention hits ~50% and a bi‑linear baseline 25%. Shows a three‑stage gradient‑descent trajectory recovers the task and selects relevant tokens with linear sample complexity, a phenomenon related to hardening of softmax. Empirically on noisy CIFAR embeddings and Cryo‑ET particle picking, prompt‑tuning gives better or on‑par accuracy with far less fine‑tuning data and faster runs.
AI-written summary · about
chapters (10)
- 0:00Introduction to prompt tuning
- 1:41Prompt tuning mechanisms
- 4:51Prompting in language and vision
- 8:12Prompt attention architecture
- 11:42Contextual data modeling
- 15:10Expressivity and theoretical results
- 20:18Optimization and generalization
- 22:53Empirical evidence and applications
- 24:52Cryo-ET research application
- 26:57Proof analysis and conclusion
related talks
36mML Efficiency for Large Models: From Data Efficiency to Faster Transformers
01h 25mPaper Discussion: Attention is all you need
30mIs Prompt Engineering Dead? How Auto-Optimization is Changing the Game
01h 30mToward Understanding In-context Learning
20mMelanie Warrick, Deep Learning Engineer, Skymind.io
26mA Shallow Introduction to Self-Attention
42mAttention and Memory in Deep Learning Networks
40mPanel discussion "NLP"