Is that LLM feature any good?
summary
Shows that an R port of InspectAI called Vitals lets R chat apps built with Ellmer be evaluated end to end: dataset (user queries and targets), solver (Ellmer client with RAG), and scorer (LLM-based grading). A conference schedule chatbot (20–30 lines of R) runs eight test queries in ~11–12 seconds and scores six of eight as correct; failing cases appear in a Log Viewer so engineers can drill into why.
AI-written summary · about
chapters (7)
- 0:00Introduction to the conf bot
- 2:38The need for LLM evaluation
- 4:00The importance of rapid iteration
- 6:36Building evaluation datasets
- 8:53Implementing solvers and scorers
- 11:51Running evals with Vitals
- 13:50Q&A and concluding thoughts
related talks
34mEvaluating LLM Frameworks
31mBeyond LLMs and Maturing RAG use cases
01h 34mHow to Construct and Assess Your First LLM Based RAG System
24mRubyLLM: Making AI Development Beautiful Again
24mDemocratizing data access through LLM
50mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
01h 49mAI engineering with open access LLMs that lie, curse, and steal
19mTesting LLM-Powered Applications