Validating LLMs – A True Story
summary
Built an automated validation framework that feeds hundreds of thousands of Java test cases into a code assistant and scores outputs weekly, combining Watson X code assistant results, code- and text-similarity (BLEU/ROUGE/code-BLEU), sentence embeddings, code coverage, Checkstyle, and a calibrated LLM-as-judge. OpenRewrite recipes run for Java 8→21 modernization and assisted fixes use edit-distance gating to require deterministic patches. Dashboard marks >60% green, 50–60% caution, <50% fail; the pipeline runs reliably and exposes preprocessing bugs in competing models.
AI-written summary · about
chapters (7)
- 0:00Introduction and background
- 4:03Challenges testing AI apps
- 17:46Building validation framework
- 21:44Metrics and evaluation methods
- 36:10Combined scoring approach
- 42:15Framework results and impact
- 45:21Q&A and availability
related talks
25mEvaluating An LLM's Ability To Code
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
35mEvaluating LLM systems: the nuts and bolts
38mMaking LLMs reliable - A practical framework
33mPractical Evaluation of LLMs for Code
39mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
29mHow to roll your own LLM