How OpenTelemetry Helps Generative AI
summary
Shows how OpenTelemetry tracing, plus logs for large prompts, turns inscrutable LLM pipelines into measurable systems: a production feature built with GPT-3.5 in production logs a pipeline of 38 upstream operations and about two dozen post‑processing steps, uses traces correlated with prompt/input/output to find classes of bugs, and drives evaluations that improve quality. Observability cuts mean response times drop from ~5s to 1.5s average, and the LLM Semantic Conventions WG and SDK wrappers aim to standardize span events and redaction.
AI-written summary · about
chapters (5)
- 0:00Introduction to AI challenges
- 1:01Reliability in LLM apps
- 10:58Observability as a solution
- 11:22Using OpenTelemetry for AI
- 17:10Advanced debugging and future
related talks
29mThe Hater's Guide to Dealing with Generative AI
31mDeep Dive To AI Agent Observability
25mThe Story of OpenAI GPT3 - Improve Observability with Tracing
18mWhat did AI just do? Making LLMs Observable with OpenTelemetry
40mOptimizing AI Applications for Production with Observability
41mGenerative AI: bliss or miss?
16mWhat Did the AI Just Do?: Making LLMs Observable with Open Telemetry
32mEnd-to-End Observability for LLM Inference: From Token To GPU