Ensuring Reliable Evaluation Systems for Your ML
summary
Concrete, actionable steps for turning vague benchmarks into trustworthy evaluation: define a contract-aligned evaluation suite tied to specific user intents, assemble a golden dataset of targeted edge cases and stratified slices, enforce inter-annotator agreement thresholds and audit label drift, instrument continuous monitoring and scheduled relabeling, and plug evaluation into the CI/CD pipeline so metrics gate promotions with clear pass/fail thresholds and automated slice analysis.
AI-written summary · about
chapters (8)
- 0:00Introduction and conference praise
- 2:28Agentification of software explained
- 9:21UX challenges with current agents
- 18:33Beyond chat interfaces for agents
- 24:59Gateway of the internet battle
- 25:45Trust issues in agentic systems
- 32:51Potential future of agentic marketplaces
- 34:40Closing remarks and Q&A
related talks
41mFrom Lab To Life: Practical AI System Evaluation
20mFive hard earned lessons about Evals
14mImproving Large Language Model Performance
27mModel Evaluation for Humans
30mHow to Properly Test ML Models & Data
19mApples, Oranges, and ML Models: Model Validation vs Benchmarking
33mThe Challenge of Valid Evaluations
27mThe Science and Practice of Open and Scalable LLM Evaluations