Finding the Needle in the Haystack or Troubleshooting Distributed Systems
summary
Claims that cross-language tracing fixes hard-to-find application errors by attaching two injected IDs to a request, logging structured JSON events, and shipping them via UDP multicast or HTTP to a central store. Implements LS for lightweight events, Mon/monemand for aggregation, a simple web UI showing per-request state, and a replay tool request replay (Queasy) that recreates failures for QA. QA uses traces daily and almost every bug report in the last two years includes a trace, and the system proves invaluable in locating data, replication, and logic faults.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:472 Tier Clustered
- 2:543 Tier Clustered
- 4:08Troubleshooting
- 7:59A Possible Solution
- 8:39Evolution of a Solution
- 10:092 Tier Distributed
- 10:55Tools Available
- 13:24Lessons Learned
- 14:32Content Match
- 16:20Use Cases
- 18:44But what about customer support? • Customer support couldn't use the command line tool • traces turned on for some number of
- 20:47Display Advertising
- 21:16N Tier FTW!
- 26:39Erlang
- 29:22Demo of UI
- 30:04Mondemand Trace Console
- 34:44Final Thoughts
- 35:49Limitations/ Future Work
related talks
57mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems
21mLightning Talk: Distributed Tracing Without the Pain!
32mA Picture is Worth 1,000 Traces
24mObservability across serverless asynchronous managed services
24mDistributed Tracing : New DevOps Foundation.
34mWould You Like Some Tracing With Your Monitoring?
29mDistributed Request Tracing
50mThe hidden gems of distributed tracing