Compute-Scaling Methods at Inference Time
summary
Compares specific inference-time methods and their tradeoffs using a quantized Llama 3 2.1B 4-bit model on Colab with 50 reasoning prompts, evaluated by BERTScore and BLEU. test-time preference optimization yields the highest accuracy but averages ~1,200 tokens per query and costs ~USD 37 per million tokens in the author’s estimate; chain of draft matches chain-of-thought accuracy at much lower token cost; a simple iterative NL2SQL loop and a step-back-to-leap-forward planner improve correctness but increase latency.
AI-written summary · about
chapters (23)
- 0:00Introduction to talk
- 0:22Speaker introduction
- 1:08Why reasoning models matter
- 2:11What is reasoning in LLMs
- 3:13Motivation for inference scaling
- 3:59Journey of large language models
- 5:05Training vs inference scaling
- 6:33What is inference time scaling
- 7:18Human analogy for inference scaling
- 7:59Models using inference techniques
- 8:44History of inference scaling
- 9:27Benefits of inference scaling
- 10:12When training is better
- 11:15Use case with SQL queries
- 12:39Chain of Thought prompting
- 15:48Chain of Draft technique
- 17:40Step Back to Leap Forward
- 19:22Test-Time Preference Optimization
- 21:50Experiment setup details
- 23:39Results and scoring metrics
- 24:45Cost vs accuracy trade-offs
- 25:50Recommendations summary
- 26:52Closing and Q&A
