Use ARTKIT to Automate and Scale Up Your LLM Evaluation Process
summary
BCG claims ARTKIT automates and scales LLM evaluation and ships as an open-source Python package that wraps vendor and custom models with asynchronous model connectors and writes calls to sqlite caching so you avoid repeat API spend. Examples include rephrasing prompts at scale, running parallel pipelines, and using LLMs as evaluators; a deployed case finds a ~5% language‐switch failure that bypasses system prompts. The framework supports hybrid human+automated testing and is explicitly a scaffolding not a one‑button verifier.
AI-written summary · about
chapters (6)
- 0:00ARTKIT introduction
- 3:13LLM deployment challenges
- 10:41ARTKIT evaluation cycle
- 23:51Automating LLM testing
- 34:20Scoring LLM outputs
- 37:12Q&A
related talks
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
29mGenerative-AI: Usecase-Specific Evaluation of LLM-powered Applications
35mEvaluating LLM systems: the nuts and bolts
32mA Practical Guide to LLM Evaluation
34mEvaluating LLM Frameworks
38mMaking LLMs reliable - A practical framework
44mPractical LLM Evals: Down-to-Earth Tooling for Everyone
29mCustomizing and Evaluating LLMs, an Ops Perspective