Tracing AI Calls End to End: What AI inference teaches us about...
summary
Shows that gRPC transport-level metrics miss critical AI inference signals: two requests with nearly identical call durations (∼30 ms apart) have time to first message values 18× different, and KV handoffs between prefilling and decoding break trace continuity in 4/11 tested transfers. Uses gRPC + OpenTelemetry, a public repo with reproducible demos, and argues for adding inter-message gap and TTFM histograms at the transport boundary so teams can correlate gRPC method/target/status with model-level attributes instead of relying on brittle application-layer wrappers.
AI-written summary · about
chapters (12)
- 0:00Introduction to tracing AI calls
- 1:09Prefill vs decode in inference
- 3:16Time to first message matters
- 6:51gRPC observability gaps
- 10:44Disaggregated inference flow
- 12:07Trace ID transfer failures
- 13:30OpenTelemetry for GenAI
- 14:55Missing shared attributes
- 16:44Proposed gRPC metrics
- 17:48Open source tools and repo
- 19:39Exemplar sampling issues
- 20:22Q&A and closing remarks
related talks
32mEnd-to-End Observability for LLM Inference: From Token To GPU
40mOptimizing AI Applications for Production with Observability
26mAI Traffic Is Different: Rethinking Proxy Arc...
18mgRPC and AI
31mDeep Dive To AI Agent Observability
26mShould You Use gRPC in Your AI Tooling Stack?
32mUnified Observability in GRPC: Metrics and Tracing Using OpenTelemetry Plugin
30mMastering Distributed Tracing with Go