Responsible AI with fmeval - an open source library to evaluate LLMs
summary
Open-source fmeval provides a structured pipeline of data loaders, model runners, evaluation algorithms and reports to move LLM work from experiments to production, claiming factual QA, summarization, generation and classification support. Demo runs a QA notebook with question/answer pairs, emits per-example 1/0 scores and JSON/pandas outputs, uses Ray acceleration to parallelize jobs (≈1.07s per chunk, ~3 minutes end-to-end) and offers SageMaker/Bedrock/HuggingFace runners plus plotting for stakeholder-ready results.
AI-written summary · about
chapters (9)
- 0:00Responsible AI with fmeval
- 1:57LLM evaluation lifecycle
- 5:28AI pillars: privacy, security, safety
- 7:41Metrics for Q&A tasks
- 10:50fmeval library structure
- 12:36Demo: fmeval evaluation
- 17:14Prepare, evaluate, review results
- 18:19Contribute to open source
- 19:02Q&A
related talks
32mStrategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop)
01h 28mMastering RAG Evaluation with MLflow & Open Source LLMs
42mTesting What Matters: Evaluating LLM Relevancy with DeepEval
24mHow to Test AI Features Before Your Users Do
41mFrom Lab To Life: Practical AI System Evaluation
46mEnhancing Agent Reliability Through Evaluation-Driven Development
26mAI Showdown: Open Source Tools for LLM Face-Offs
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work