~/talk

Compute-Scaling Methods at Inference Time

Talkat PyCon India 2025 · 2025-09-12..09-15· [large-language-models]
linkedin:agnitin
slides:none on file

Compares specific inference-time methods and their tradeoffs using a quantized Llama 3 2.1B 4-bit model on Colab with 50 reasoning prompts, evaluated by BERTScore and BLEU. test-time preference optimization yields the highest accuracy but averages ~1,200 tokens per query and costs ~USD 37 per million tokens in the author’s estimate; chain of draft matches chain-of-thought accuracy at much lower token cost; a simple iterative NL2SQL loop and a step-back-to-leap-forward planner improve correctness but increase latency.

AI-written summary · about

  1. 0:00Introduction to talk
  2. 0:22Speaker introduction
  3. 1:08Why reasoning models matter
  4. 2:11What is reasoning in LLMs
  5. 3:13Motivation for inference scaling
  6. 3:59Journey of large language models
  7. 5:05Training vs inference scaling
  8. 6:33What is inference time scaling
  9. 7:18Human analogy for inference scaling
  10. 7:59Models using inference techniques
  11. 8:44History of inference scaling
  12. 9:27Benefits of inference scaling
  13. 10:12When training is better
  14. 11:15Use case with SQL queries
  15. 12:39Chain of Thought prompting
  16. 15:48Chain of Draft technique
  17. 17:40Step Back to Leap Forward
  18. 19:22Test-Time Preference Optimization
  19. 21:50Experiment setup details
  20. 23:39Results and scoring metrics
  21. 24:45Cost vs accuracy trade-offs
  22. 25:50Recommendations summary
  23. 26:52Closing and Q&A
~/search
type to search 6,518 conferences, 143,918 speakers and 192,133 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
← →
on a talk: 5 seconds back / forward
0–9
on a talk: jump to 0%–90% of the recording
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.