Your Prompt Is Not a Unit Test — instrumenting LLM apps in production
summary
Shows how to instrument LLM features end to end: capture OpenTelemetry spans and trace UI → agent → Redis/Postgres → model calls (Azure OpenAI, Bedrock) to measure token counts, per-model cost, and latency, and to count actual LLM calls versus internal spans. Covers sampling controls, prompt versioning tied to eval drops, and using an LLM-as-judge plus human thumbs-up to detect silent failures like hallucinations. Demoed a live Infrastructure Advisor AI pipeline with tool calls to ArcGIS and real-time guardrails blocking prompt injections.
AI-written summary · about
chapters (15)
- 0:00LLM observability challenges
- 1:31LLM app failures vs traditional apps
- 3:40Metrics for LLM apps
- 5:28Collecting AI app data
- 7:13Observability in LLM frameworks
- 9:23Sampling and cost control
- 10:27Evaluating LLM outputs
- 11:50Detecting LLM failures
- 14:17Tuning prompts and tools
- 15:22Prompt versioning impact
- 17:08Validating changes with data
- 18:54Monitoring adoption and feedback
- 20:40Demo: Infrastructure Advisor AI
- 23:49Real-time monitoring and costs
- 24:33Q&A
related talks
01h 00mPrompt Failures and Latency Spikes: Observability for AI
48mBeyond the Prompt: Evaluating, Testing, and Securing LLM Applications
16mWhat Did the AI Just Do?: Making LLMs Observable with Open Telemetry
24mHow OpenTelemetry Helps Generative AI
26mMastering Large Language Models: Prompt Engineering for Precision & Performance
01h 26mShipping AI That Works: An Evaluation Framework for PMs
43mBuild and Maintain your Company Copilot with Azure ML and GPT-4 Q&A
01h 39mMastering LLMs: From Prompt Engineering to Agentic AI