StriaTrace: Efficient Tracing and Diagnosis for Online LLM Inference
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
StriaTrace implements host and device collectors that record only 13 synchronization spans and kernel boundaries plus memory transfers, with a pass-by sampler every 10 ms and adaptive regression-based roofline filtering that retains roughly 1.5% of steps for long-term storage. It cuts tracing traffic and overhead (97.8% less than heavy profilers), keeps host overhead <1% and GPU overhead from ~5% down to 0.6% as throughput scales, and surfaces correlation evidence that diagnoses hundreds of anomalies across 19 root causes—e.g., rewriting a hot Python op as a C++ custom op drops a sampled step from ~160 ms down to ~43 ms.
AI-written summary · about
chapters (13)
- 0:00StriaTrace for LLM inference
- 1:21Diagnosing LLM inference latency
- 2:44Why existing tracers fail
- 4:27Design principles of StriaTrace
- 5:10StriaTrace architecture
- 5:52Host-side instrumentation
- 6:56Pass-by sampling
- 7:42Device-side instrumentation
- 8:25Anomaly detection
- 9:09StriaTrace overhead evaluation
- 10:14Production diagnosis examples
- 10:38Token sampler latency spike
- 11:21Six months in production
related talks
37mKernel and userspace tracing with LTTng and friends
48mEfficient and Large-scale Infrastructure Monitoring With Tracing
05mPerformance analysis using Systrace
35mTracing is More Than Traces: The Insights in Trace Aggregates
42mHow Much Is Tracing? Measuring the Overhead Caused by the Tracing Infrastructure
25mPerfect Match: Correlating Continuous Profiling with Distributed Tracing for Stronger Observability
21mKUtrace For RISC-V
57mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems