Building Reliable AI Agents for Publishing: A DSPy-Based Quality Assurance Framework
summary
A newsroom-grade pipeline built on DSPy composes modular text transformation graphs to run multistep generation (chain-of-thought), automatic knowledge-graph factuality checks (Neo4j), multiple LM judges and role-based personas, plus human-in-the-loop review. The deployed quality gate handles about 13,000 requests per month, uses canary deployments and monitoring, targets 3–4 sigma quality, and balances prompt optimization, occasional fine-tuning, and strict low-latency constraints.
AI-written summary · about
chapters (10)
- 0:00Introduction and challenges
- 3:06LLM evaluation techniques
- 8:27Advanced judge models
- 10:14Personas in evaluation
- 10:58DSPY framework basics
- 18:42Prompt optimization
- 19:20Quality gate architecture
- 20:24Monitoring and KPIs
- 21:14Real-world best practices
- 23:50Q&A session
related talks
35mThe Podcast Pipeline Nobody Sees: Behind the Scenes with AI Agents and AI gen
57mEvaluating AI Agents in production with Python
26mDSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
09mRethinking AI: Building What Matters, Not Just What's Possible
31mDSPy: The End of Prompt Engineering as We Know It
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
34mScaling Trust, Not Automation - Rethinking Quality for the AI Era
28mWhy Software Quality Is the Key to Scaling AI Development