From Chaos to Control: Building Enterprise-Grade LLM Agents That Actually Work
summary
Claims that 99% per-step accuracy still yields ~18% failure over 20 steps and 95% yields nearly two-thirds failures drive concrete fixes: use the largest model first, allocate test-time compute for multi-pass reasoning, split workflows into independent subcomponents, and enforce detailed open API-like specifications for tools. Evaluation practices include per-step AST and execution tests, thresholds for mitigation (under 10%: prompt work, 10–20%: targeted annotations, above: fine-tune/post-train), and a demoed North finance agent that shows a reasoning trace with citations and controllable tool access.
AI-written summary · about
chapters (16)
- 0:00Coher and enterprise AI
- 1:442025 pivotal moment for AI agents
- 3:09AI agents and core principles
- 3:55Evolution of AI systems
- 5:20Enterprise challenges deploying AI agents
- 7:06Real-world agent workflow failure
- 8:51Large models to combat complexity
- 9:57Test time compute for reliability
- 11:03Breaking tasks into units
- 11:47Context limits in LLMs
- 13:15Tool interface challenges
- 15:42Multi-tool agent ecosystems
- 16:45Importance of agent evaluations
- 19:55Mitigating agent failures
- 20:57Key takeaways for enterprise agents
- 21:39Demo: Coher's North platform
related talks
46mEnhancing Agent Reliability Through Evaluation-Driven Development
50mHow to Test LLM Agents
56mHerding LLMs: Predictable Outcomes from Unpredictable Agents
29mBuilding Trust in AI Agents
44mProduction-Grade LLMs & AI Agents
02h 22mArchitecting and Testing Controllable Agents
41mBuilding LLM Agents – Challenges and Learnings
15mThe 85% Beyond the Model