Resolving Outages Faster with Better Debugging Strategies
summary
Claims that replacing static dashboards with on-the-fly query editing and trace-linked samples cuts mean time to resolve: use a query editor to layer peel from a region 98th‑percentile SLI (100 ms target) down to a single replica, confirm a broken instance (kill it to mitigate), then use dynamic data joins to correlate latency with kernel version and find stragglers. Add tracing exemplars (sampled synthetic RPCs ~1/100) to jump straight to distributed traces — the demo uses Panopticon and trace-linked exemplars.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:15Introductions
- 0:59A note about technologies
- 4:12The debugging feedback loop
- 5:39How can we improve resiliency?
- 6:31Error Percentage of slow user queries too high
- 7:52Layer Peeling
- 11:28Back to bisecting
- 20:36Dynamic Data Joins
- 28:14Exemplar Tags
- 29:59Exemplar Traces
- 33:18System Diagram
- 34:19Exemplar Trace Collection Code
- 34:41Exemplar Client-side Tag Collection Code
- 35:03Exemplar Deduplication Code
- 35:22Exemplar Cardinality Sampling Code
- 35:323. Exemplars
- 36:45Post-incident cleanup
- 37:16Summary
related talks
28mResolving Outages Faster With Better Debugging Strategies
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...
39mHow to Use Common Investigation Practices in Troubleshooting
19mBest Practices for Rapid Insights from Metrics, Logs, Traces and Profiles
07mWhy Tracing is a Lifesaver for E2E Tests of Distributed Systems?
13mOptimizing System Performance: O11y in Distributed Environments
01h 01mThe Performance Investigator's Field Guide
35mTracing is More Than Traces: The Insights in Trace Aggregates