The Sorcerer’s Apprentice: The craft of building reliable solutions with LLMs
summary
LLM prototypes often fail on real data because models are non‑deterministic, can’t count, and have short corruptible memory; reliability requires pruning output space and closed‑loop feedback. Capture full traces, label failure modes with domain experts, run build–measure–learn vertical slices, add binary input/output validators as CI regression tests, and monitor live with guardrails (rate limits, masking, human‑in‑the‑loop). Pair that with threat modeling (assets, attack surface, abuse cases, mitigations) and defense‑in‑depth to make LLMs predictable and safe.
AI-written summary · about
chapters (6)
- 0:00Building reliable LLM solutions
- 2:29Sorcerer’s Apprentice lessons
- 8:07LLMs: powerful but unpredictable
- 14:03Lean data science for LLM uncertainty
- 20:39Threat modeling for LLM reliability
- 30:14Key principles for reliable LLM solutions
related talks
47mHarnessing LLMs for Next-Level Security Analysis
38mMaking LLMs reliable - A practical framework
15mKeynote: Into the Black Box: Observability in the Age of LLMs
46mEnhancing Agent Reliability Through Evaluation-Driven Development
35mEvaluating LLM systems: the nuts and bolts
47mA Step-by-Step Guide to Securing Large Language Models (LLMs)
44mI Abuse Your LLM Live with 10 Security Vulnerabilities
31mWhat problems should enterprises actually trust an LLM to solve?