Keynote: The neglected art of a relevant benchmark and why you need one
summary
Argues that useful benchmarks save wasted compute and wrong projects, with concrete examples: teams launched 8 AI products for millions of users, training runs take 24 hours to 4 days and cost thousands, and a recruiting shortlist target matches 80% overlap between two humans. Shows how to define the actual task (shortlist not hire), estimate usefulness (capture ~80–90% of human picks), and warns that LLMs like Claude and search tools (Perplexity OpenAI) complicate but don’t replace rigorous benchmarks.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 4:19Python in AI development
- 8:17Defining AI benchmarks
- 13:39Performance benchmarks for usefulness
- 20:07Benchmarks prevent wasted effort
- 22:13Real-world task benchmarking
- 28:30Generative AI benchmark challenges
- 32:00Tips for benchmarking
- 35:10Q&A
related talks
16mBenchmarks Are Memes: How What We Measure Shapes AI—and Us
54mKeynote: Can you trust your (large language) model?
30mBuilding Teams and Culture that Support ML Innovation
32mKeynote: AI & the stuff built for AI - are they actually useful for data science?
36mRun a benchmark they said. It will be fun they said.
36mKeynote: AI, ML, Cloud for ML
19mApples, Oranges, and ML Models: Model Validation vs Benchmarking
05mInterview