A Year of Observability Across the Toolstack: Breakthroughs, Challenges, Improveme...
summary
The project uses distributed tracing for over a year to produce real‑time activity trees across SDK, xen-api, xenopsd and Python glue, showing exact call chains and a flame view that pinpointed storage hotspots (VDI activate/attach) as the dominant VM start latency. An observer.py wrapper instruments arbitrary Python commands, and a BPFTrace prototype samples the datapath during live migrations. Data exports use W3C traceparent propagation, which works when headers travel but remains the main integration challenge; CLI tooling and infra‑as‑code exist but automated pass/fail analysis is still incomplete.
AI-written summary · about
chapters (15)
- 0:00Introduction and background
- 1:51Core observability challenges
- 3:56Features and environments in observability
- 6:03Logs vs traces vs metrics
- 8:10Observer infrastructure and endpoints
- 9:34Instrumenting the toolstack
- 11:25Extending to data path tracking
- 13:12Control plane insights and automation
- 15:18Querying and visualizing traces
- 16:44Trace context propagation
- 19:58Standards and trace parent challenges
- 21:24Analysis and infrastructure gaps
- 22:46Design patterns for observability
- 24:31Data path trace parent discussion
- 25:44Q&A session
related talks
49mImproving Observability of Distributed Systems
17mEfficient Python Observability with OpenTelemetry
29mModern observability for Python
33mThe Observant Developer - OpenTelemetry from code to prod
46mObservability in the real world: Improve systems w/iterative approach
31mObservability for Developers: How to Get There
53mUnmask hidden issues with Observability superpowers!
23mA peek into observability from a tester's lens