Keynote: Can you trust your (large language) model?
summary
Argues that LLM claims break down when measured correctly: replicated psychology finding fails, CheXNet pneumonia model succeeds only because of data leakage, GPT‑4 solves CodeForces puzzles released before its training but fails later ones, Hugging Face retires benchmarks like MMLU/ HellaSwag for poor questions, prompt tweaks cause up to 77% performance swings, and Chatbot Arena’s ELO hides vague “good” criteria. Recommends engineering practices: unit tests, trace logging, human review and A/B testing tied to concrete tasks.
AI-written summary · about
chapters (9)
- 0:00Keynote introduction
- 4:47Psychology of wrong metrics
- 9:15Reliability in ML models
- 16:19Flaws in LLM benchmarks
- 27:03Multiple-choice tests mislead
- 32:36Shift to overambitious benchmarks
- 37:33Data leakage and prompt sensitivity
- 45:43Assessing LLMs like software
- 52:24Summary and expectations
related talks
54mKeynote: Can you trust your (large language) model?
01h 04mCan you trust your (large language) model?
01h 05mCan you trust your (large language) model?
46mKeynote: How Can We Trust Machine Learning?
33mOpening Keynote: LLMs, between fantasy and reality - the poin...
48mKeynote – Sameer Singh
25mWhy LLMs Still Fail: Reasoning, Robustness, and Safety
20mThe State of Truth and Trust at Work