From Traces To Action: Auto-Instrumenting LLMs for Observability...
summary
Demonstrates auto-collecting traces, metrics, logs from an LLM app using OpenTelemetry auto-instrumentation, enriching requests with model, token and latency attributes, and shipping them through a collector into Data Prepper for indexing into OpenSearch 2.11; demo shows live traces and token counts lighting up in dashboards, requires custom index templates and occasional manual spans for deep context, and recommends adding eBPF profiling and OpenLIT SDK for per-span CPU and cost breakdowns.
AI-written summary · about
chapters (11)
- 0:00LLM observability challenges
- 3:14Why observability matters
- 6:48OpenTelemetry and its role
- 8:13OpenTelemetry capabilities
- 11:21OpenSearch integration
- 15:55Data flow to dashboards
- 22:41Live auto-instrumentation demo
- 24:52Analyzing traces in OpenSearch
- 27:17LLM observability use cases
- 27:59Best practices for observability
- 30:09Future tools and takeaways
related talks
26mPlug-and-Play Observability: OTel Auto-Instrumentation for LLMs in OpenSearch
19mExamining the Principles of Observability and Its Relevance in LLM...
23mModern Application Debugging: An Intro to OpenTelemetry
18mWhat did AI just do? Making LLMs Observable with OpenTelemetry
32mOpenTelemetry feat. logs, metrics and traces
36mObservability For You and Me with OpenTelemetry
47mUsing metric and traces to diagnose our cloud-native applications
50mGuide to Observability with OpenTelemetry