How to evaluate your Evals?
summary
Marto AI’s ATO platform runs evals on an Appmaster support bot using deterministic regex checks for a JSON schema plus vector similarity and a model-based judge, with test data including edge cases. A live notebook experiment shows initial failures on schema and grounding, iterative prompt and instruction changes plus swapping models (picked "clo") produce improved outputs and a reported 100% success rate across the defined evals, demonstrating measurable prompt engineering and deployable results.
AI-written summary · about
chapters (8)
- 0:00Introduction to Marto AI and ATO Platform
- 0:37Understanding Evals in Gen AI
- 1:16Challenges in AI Testing
- 2:23Example Use Case: Technical Support Bot
- 4:03Types of Evals
- 5:31Implementing Evals in Practice
- 8:44Live Experiment and Results
- 12:18Conclusion and Next Steps
related talks
24mHow to Test AI Features Before Your Users Do
20mFive hard earned lessons about Evals
15mEvals Are Not Unit Tests
47mBuild Trustworthy AI Agents powered by Evals
43mEvaluation is all you need
15mPractical tactics to build reliable AI apps
46mEnhancing Agent Reliability Through Evaluation-Driven Development
41mTaming Testing of AI apps