How to Tell If Your Agent Used the Right Stuff
summary
Claim: most agent "failures" are actually failures of context delivery, not model capability. Workflow: instrument traces, save rewritten queries, retrieval params, retrieved chunks and metadata, log tool calls and confidence scores, then run attribution and citation checks and targeted probes. Surprising finding: models often ignore large context — 40% of the context window unused in tests — and developer instructions sometimes hurt; best result came with no extra instructions. Measure retrieval faults (missing recall, bad ranking, distractors, stale docs) and use a lightweight golden dataset for repeatable regression checks.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:10Single turn LLM applications
- 7:47Baseline understanding of models
- 15:06Single agent failures
- 21:13Agent system design
- 28:16Evaluation and testing
- 34:19Start with evaluation
- 39:58Fixing and iterating
- 40:45Q&A
related talks
26mWhy More Context Makes Your Agent Dumber and What to Do About It
01h 20mMind the Gap (In your Agent Observability)
33mTrust but Verify: Testing Agents in Copilot Studio
25mFrom Chaos to Control: Building Enterprise-Grade LLM Agents That Actually Work
45mBuild Your Own Benchmark: Why Agents Need Custom Evals and How to Create Them
54mYou Can't assertEquals an Agent: A Tester's Guide to Agentic Quality
08mHow to Improve your Vibe Coding
02h 04mShip Real Agents: Hands-On Evals for Agentic Applications