How can I trust you AI and the leap of faith
summary
Claims that evaluation is the bottleneck for production AI, with a concrete pipeline for extracting sustainability data from PDFs, swapping parsers (using Adobe API) and raising extraction accuracy from 73% to 80%. Shows practical failures: hallucinations, nondeterministic outputs, misread tables and infographics. Recommends AI-as-judge plus human alignment checks, bootstrap confidence intervals to compare evaluators, and sample size rules of thumb (300–1,000 samples) to make evaluation results reliable.
AI-written summary · about
chapters (10)
- 0:00Trusting AI systems
- 1:51AI evaluation role
- 3:16Evaluating AI output challenges
- 6:11AI evaluation techniques
- 9:29Practical evaluation pipeline
- 13:22System components and success
- 16:53Reliable evaluation
- 19:42AI and human evaluations
- 21:31Statistical confidence in results
- 24:21Conclusion and future
related talks
46mCan you trust your AI?
26m7 Habits of Highly Effective Generative AI Evaluations
46mEnhancing Agent Reliability Through Evaluation-Driven Development
31mMeasurement of Trust in AI
24mHow to Test AI Features Before Your Users Do
15mPractical tactics to build reliable AI apps
38mAI Just Happened to Me : Help!!
57mEvaluating AI Agents in production with Python