The Future of Language Models and Transformers 2025
2025-03-31..2025-04-04
Berkeley, United States
who it's for
language model researchers, research engineers, inference and deployment engineers, evaluation and benchmark scientists, reinforcement learning researchers, multimodal and vision-language researchers, privacy and federated learning engineers
topics
retrieval-augmented vs long-context models; MoE-aware federated LM training; policy optimization for LLMs (PPO++, REBEL); activation-free sequence-to-sequence blocks; long-chain internal reasoning and rumination; clone-structured causal graphs for navigation; phrase-object vision-language alignment; latent diffusion for text generation; inference-time compute scaling; separability-weighted evaluation and dataset pipelines; spectral dimension reduction for transformers; learning generative models from corrupted data
| title | author | length |
|---|---|---|
| The Frontier between Retrieval-augmented and Long-context Language Models | Danqi Chen | 01h 00m |
Presentation argues that retrieval-augmented models still lack a reliable training recipe because joint updating of encoder and datastore forces costly… | ||
| Federated Open Language Models: Training LMs on Distributed Data | Sewon Min | 58m |
SILO Open LM trains a 7B transformer on public Common Crawl (1T tokens) then has each data owner do isolated continual pretraining (≈50B tokens) using… | ||
| Efficient Policy Optimization Techniques for LLMs | Kianté Brantley | 46m |
Introduces three concrete algorithms: PPO++ mixes roll‑ins from a reference policy to boost exploration and shows a small GPT‑4 win‑rate gain over PPO; DR‑PO… | ||
| Towards sequence-to-sequence models without activation functions | Grigorios Chrysos | 56m |
Argues that sequence-to-sequence models can drop element-wise activations and get expressivity from explicit polynomial expansions implemented with Hadamard… | ||
| DeepSeek-R1 Thoughtology: ＜Thinking＞ about LLM Reasoning | Siva Reddy | 01h 00m |
DeepSeek R1 produces long internal “thoughts” with an initial Bloom step that finds a first solution and repeated Reconstruction cycles (reblooms and… | ||
| On cognitive maps, LLMs, world models, and understanding | Dileep George | 01h 06m |
Claims that space is a latent sequence and that hippocampal “place cells” reflect sequence-conditioned latent states rather than GPS-like locations; introduces… | ||
| The Future of Language Models: A Perspective on Evaluation | Swabha Swayamdipta | 01h 06m |
Argues that evaluation lags model capability and gives concrete methods and results: builds a human-labeled social-media dataset of 4,000 annotated tweets and… | ||
| Attention to Detail: Fine-Grained Vision-Language Alignment | Kai-Wei Chang | 46m |
Presents concrete methods and results: adapts CLIP-style models into phrase-object alignment using Visual Genome (≈1,000 categories, millions of boxes) plus… | ||
| The Move Toward AGI: Why Large Language Models Surprised Almost Everyone... | Theoretically Speaking | Anil Ananthaswamy, Dileep George, Azalia Mirhoseini, Luke Zettlemoyer | 01h 11m |
Discussion gives concrete claims: scaling laws from 2020 reliably predict pretraining progress; in-context learning (zero/one/few-shot) emerges from next-token… | ||
| Transformers can learn compositional function | Jason Lee | 01h 04m |
Two concrete results: a shallow single-index teacher-plus-student model shows sharp d^{k-1} plateau emergence times (Hermite coefficients control tiny initial… | ||
| Advancing Diffusion Models for Text Generation | Kilian Weinberger | 01h 02m |
Introduces Latent Diffusion for Language Generation and a pretraining recipe called LMLM that replaces tail facts with fake DB lookup tokens and forbids… | ||
| Inference Scaling: A New Frontier for AI Capability | Azalia Mirhoseini | 01h 08m |
Claims that repeated sampling produces predictable gains: coverage (pass@k) scales log-linearly with samples across four orders of magnitude and fits an… | ||
| Predicting and optimizing the behavior of large ML models | Andrew Ilyas | 01h 04m |
Presents a concrete method that differentiates the final model performance with respect to pre‑training choices by computing a meta gradient through training… | ||
| Mixed-modal Language Modeling: Chameleon, Transfusion, and Mixture of Transformers | Luke Zettlemoyer | 01h 06m |
A unified early-fusion effort tests three concrete systems and training facts: Chameleon (tokenized images) converts images to ~1,000 discrete tokens and hits… | ||
| Reducing the Dimension of Language: A Spectral Perspective on Transformers | Elad Hazan | 01h 00m |
Argues that sequence data can come from high‑dimensional linear dynamical systems yet admit a low‑dimensional spectral representation (about 30 effective… | ||
| LLM skills and meta-cognition: scaffolding for new forms of learning? | Sanjeev Arora | 01h 03m |
Shows that frontier models actually extract and name procedural skills and can compose them: GPT-4o reliably generates novel skill labels and composes up to… | ||
| The Key Ingredients of Optimizing Test-Time Compute and What's Still Missing | Aviral Kumar | 01h 02m |
Formalizes optimizing test‑time compute as meta reinforcement learning and measures success by cumulative regret over long output streams, not just final 0/1… | ||
| Controllable and Creative Natural Language Generation | Nanyun (Violet) Peng | 01h 01m |
Argues that creativity requires constraint-aware search and shows three concrete results: lyric generation (Lyra) uses a hierarchical plan plus a modified beam… | ||
| Learning Generative Models from Corrupted Data | Gianis Daras | 01h 09m |
Proposes training diffusion generative models from corrupted examples by learning denoisers via the score–conditional expectation identity, starting with… | ||
| nothing matches | ||