The Benefit of Hindsight: Tracing Edge-Cases in Distributed Systems
summary
Proposes retroactive sampling and an implementation, Hindsight, that traces every request into a fixed-size buffer pool in shared memory but only ingests traces after a programmatic symptom triggers collection. Claims nanosecond-level trace APIs, handles GB/s per node and millions of requests/sec, and captures nearly all rare edge-case traces while adding ≈3.5% peak throughput overhead (vs no tracing) — versus tail sampling that causes much higher latency and throughput loss.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:26Distributed Applications
- 2:29Distributed Tracing
- 5:23Sampling vs. Edge-Cases
- 6:10Observations
- 7:20Retroactive Sampling
- 9:05Hindsight
- 12:30Evaluation
- 13:21Overhead with 100% Requests
- 14:27Overhead vs. Edge-Cases
- 15:28Conclusion
related talks
14mCognitive and SelfAdaptive System for Distributed Tracing
49mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems
21mLightning Talk: Distributed Tracing Without the Pain!
11mCognitive and Self-Adaptive System for Effective Distributed-Tracing...
41mTracing Real-Time Distributed Systems
33mReal-World Sampling – Lessons Learned...
13mCognitive and Self-Adaptive System for Effective Distributed-Tracing
41mFinding the Needle in the Haystack or Troubleshooting Distributed Systems