The Key Ingredients of Optimizing Test-Time Compute and What's Still Missing
summary
Formalizes optimizing test‑time compute as meta reinforcement learning and measures success by cumulative regret over long output streams, not just final 0/1 accuracy. Proposes finetuning that optimizes intermediate tokens with dense progress rewards (information‑gain style advantages computed via a separate prover mu), implements MRT on distilled R1/Quinn 1.5B traces, and finds R1 chains often degrade with more thinking while self‑consistency/majority voting recovers performance. Theory and experiments claim RL with verification outperforms expert‑cloning/STaR and the gap grows with data and token budget.
AI-written summary · about
chapters (11)
- 0:00Introduction to scaling
- 6:12Desiderata and formulation
- 10:48Analyzing current models
- 17:43Majority voting analysis
- 22:00Learning objectives
- 26:15Budget-agnostic training
- 31:01Dense reward design
- 36:41Implementing dense rewards
- 47:36Experimental results
- 52:31Open questions and future
- 54:04RL vs SFT theory
related talks
39mLet LLMs Wander: Engineering RL Environments
56mA Tutorial on Reinforcement Learning II
41mLet LLMs Wander: Engineering RL Environments
34mOffline Reinforcement Learning and Model-Based Optimization
18mReward Augmentation in Reinforcement Learning for Testing Distributed Systems
02h 42m[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents
23mTTT, TAO, δ-mem: When to Spend a Gradient
11mLinear Predictors in Reinforcement Learning: ...