Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop)
summary
Walks through hands-on evals that actually run: deploys an IBM-granite model to a VLM inference container on L4 GPUs, then sweeps request rates with GuideLLM to measure latency, throughput, time-to-first-token and exportable JSON; compares mean/median/P99 tails for SLO decisions. Follows with lm-eval-harness MMLU‑Pro accuracy runs and a Promptfu safety/bias suite, shows which knobs change performance, and leaves reproducible workspaces available for further CI/CD-style integration and tuning.
AI-written summary · about
chapters (6)
- 0:00Introduction and setup
- 3:07Why LLM evaluations matter
- 15:07Hands-on system performance
- 25:57Factual accuracy with MMLU Pro
- 29:58Safety and bias evaluations
- 31:44Q&A and wrap-up
related talks
38mMaking LLMs reliable - A practical framework
44mPractical LLM Evals: Down-to-Earth Tooling for Everyone
35mEvaluating LLM systems: the nuts and bolts
27mThe Science and Practice of Open and Scalable LLM Evaluations
32mA Practical Guide to LLM Evaluation
42mTesting What Matters: Evaluating LLM Relevancy with DeepEval
29mCustomizing and Evaluating LLMs, an Ops Perspective
46mEnhancing Agent Reliability Through Evaluation-Driven Development