Observing AI Applications with OpenLit and OpenTelemetry
summary
Demo uses OpenLit on top of OpenTelemetry to instrument a TypeScript Next.js travel chatbot, sending traces, logs and metrics to a collector and Elastic. Traces show model IDs, tool calls and per-tool durations; metrics include token counts and ms-to-first-chunk for cost and latency tracking. Built-in evaluations (OpenAI/Anthropic) flag hallucinations—example: hallucination detected: Brandenburg Gate in Madrid—and guardrails catch illegal-content prompts. The live demo runs end-to-end and successfully emits alerts and evaluation verdicts.
AI-written summary · about
chapters (14)
- 0:00AI observability with OpenTelemetry
- 1:52Traditional instrumentation
- 4:41Automatic .NET and TypeScript
- 7:29Logs metrics and traces
- 17:20Why AI needs special observability
- 21:14OpenLit for AI tracing
- 23:04Demo setup for AI tracing
- 26:54Live demo AI tracing
- 31:04Evaluating AI responses
- 39:13Live demo AI evaluation
- 43:31AI safety and compliance
- 47:25Live demo AI guardrails
- 53:01AI observability summary
- 54:44Q&A on OpenLit
related talks
16mWhat Did the AI Just Do?: Making LLMs Observable with Open Telemetry
18mWhat did AI just do? Making LLMs Observable with OpenTelemetry
33mWhat Did My Agent Do? Observability and Accountability for AI Agents
30mTutorial: Building an Observable AI Agent With an Open Source Stack
40mKeep AI on Track: Guardrails + OpenTelemetry Observability
40mOptimizing AI Applications for Production with Observability
31mDeep Dive To AI Agent Observability
01h 00mPrompt Failures and Latency Spikes: Observability for AI