Evaluate and choose models in Amazon Bedrock (AIM204)
summary
Walks through a concrete LLM selection workflow using Amazon Bedrock’s Model evaluation feature to pick a summarization model for a fictional Octan University research‑paper app. Shortlist models by capabilities and context window, run inference across candidates, then run built‑in automated metrics (accuracy, robustness) and optional human judging; demo evaluates Titan Text Express (accuracy ~0.21, robustness high on one dataset). Shows how costs and latency are measured per request, how to combine statistical and model‑based metrics, and how to pick the best model for priorities.
AI-written summary · about
chapters (8)
- 0:00Introduction to Bedrock
- 3:47Defining the use case
- 6:35Model evaluation criteria
- 12:58Structuring the selection process
- 22:27Evaluation metrics and methods
- 28:55Using Amazon Bedrock features
- 31:56Evaluation demo
- 33:26Selecting the final model
related talks
39mModel choice in Amazon Bedrock
40mScaling generative AI workloads with efficient model choice
38mBuilding AI responsibly with a GRC strategy
31mLamRAG: From Data to Constructive Insights using Amazon Bedrock
22mObservability in AI-Powered Applications with Amazon Bedrock
34mBuilding Trust with LLMs
42mExplore text-generation FMs for top use cases with Amazon Bedrock
Stop Paying for Frontier Models