The Limits of Prompting: ArchitectingTrustworthy Coding Agents
summary
A prototype Baz code review agent connects to GitHub via a webhook aggregator, calls cloudset models for structured outputs, and runs concurrent benchmarks on a growing JSON/TOML dataset (aim for ~20+ examples). Initial end-to-end acceptance rates vary 12–40%; splitting work into specialized subagents and adding vector similarity and module context lifts that to over 60%. Evaluation uses a separate LLJ model, deterministic booleans where possible, and metrics that show total comments cut roughly in half while keeping hit rate stable.
AI-written summary · about
chapters (7)
- 0:00Getting Started
- 1:23Code Review Agent
- 6:40Benchmarking Insights
- 12:50Task Splitting
- 15:10Context Matters
- 21:11Awesome Reviewers
- 22:57Key Takeaways
related talks
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
50mPanel: Beyond The Vibe: Code Quality First
32mFrom Toil To Triumph: Harnessing Agentic AI To Streamline Infrastructure as Code
29mOne-Click Code Fix: Securing Code Using AI
37mAI-Driven Developer Productivity at Scale
30mThe Great Doubt: What Building an AI Agent Taught Us About Trust
45mThe AI Revolution in Software Development: From Code Writing to Quality Assurance
25mBeyond the IDE: Toward Multi-Agent Orchestration