Do you know how well your model is doing? Evaluate your LLMs
summary
Hands-on workshop that shows how to fine-tune a small GPT2 locally on GSM‑8K, run comparisons between two models (GPT2 and Qen) with Lighteval, and plug custom tasks and metrics into an evaluation pipeline; repo and scripts are provided so attendees actually download models, run three exercises, and see numbers not just claims. It reports accuracy scores, discusses safety/guard‑rails, and notes experiments sometimes succeed and sometimes fail Hugging Face Transformers custom metrics
AI-written summary · about
chapters (6)
- 0:00Introduction
- 3:52Why evaluating LLMs matters
- 10:00Fine-tuning GPT2 on math reasoning
- 13:14Comparing models with lighteval
- 21:38Custom tasks and metrics
- 25:36Closing
related talks
40mPanel discussion "NLP"
30mLearning to wield large language models
32mStrategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop)
33mBuild Better AI with First-Party Data and Fine-Tuning
19mBeat GPT 4 & Build the Best AI For You
28mFine-Tuning Small Language Models with Python
01h 04mCan you trust your (large language) model?
23mUnveiling the Journey of NLP