How to Measure Quality os LLMs
summary
A CTI-specific benchmark is under construction with five concrete use cases: summarization, named entity recognition and relationship extraction; a released tagging tool pre-labels reports at ~70–80% accuracy and asks volunteers to spend ~15–20 minutes correcting examples to build training data for LoRA or fine-tuning. Experiments show a human comparison on 50 summaries gives an average preference score of 4.7 (AI > human) while a naive TTP classifier yields F1 ≈29% that improves to ~80% after resampling and better labels, and pipelines use STIX2.1, vector DB RAG and STIX bundles for graphed outputs.
AI-written summary · about
chapters (12)
- 0:00LLM quality measurement intro
- 2:11Five CTI LLM use cases
- 5:24LLM benchmarking challenges
- 8:42CTI datasets for benchmarking
- 10:50Summarization benchmark results
- 14:28Retrieval augmented generation
- 15:51Guardrails for LLM reliability
- 19:26Semi-supervised learning loop
- 21:59Live demo: tagging tool
- 23:27Tagging TTPs with F1 score
- 24:30STIX bundles and knowledge graphs
- 25:52Key takeaways and next steps
related talks
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
44mProduction-Grade LLMs & AI Agents
34mBuilding Trust with LLMs
42mAdvancements in Evaluating Large Language Model Applications
38mThe Art of Compression: Crafting Insightful Summaries with LLMs
30mEvaluating RAGs: On the correctness and coherence of Open Source eval metrics
32mA Practical Guide to LLM Evaluation
34mScaling Trust