Techniques and Tools for a Coherent Discussion about Performance in Complex...
summary
Argues that latency comes from the spaces between services and recommends two concrete practices: distributed tracing (using zipkin/dapper tracing with sampled traces — e.g. sample 0.5% traces rather than tracing everything) to see handoffs, and high-frequency histograms stored as time-series histograms to get distributions not averages. Reports real results: an instrumentation bug fix saves ~350 ms per graph and DNS fixes save ~150 ms; shows examples like 240 million messages/day and 45 billion points over weeks to justify the scale assumptions.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:28Performance
- 3:23Ireland
- 4:00Measuring Performance
- 6:54Time
- 8:56The problem
- 10:26Space between notes
- 11:19Celebrate individual wins
- 12:12Celebrate collective wins
- 13:05The future of observability
- 16:43The web industry
- 18:46Machine induced data
- 20:40Dynamic tracing
- 22:51What does path one look like
- 25:40Why path one doesnt work
- 26:40Scribe
- 31:44Histogram
- 32:24Timeseries histograms
- 36:01Conclusion
related talks
01h 01mThe Performance Investigator's Field Guide
46mHow to Properly Blame Things for Causing Latency
32mSoftware Performance: A Shape Not a Number
01h 00mSystems @Scale Summer Q&A Week 4
54mUnderstanding Performance w/DTrace (While The Customer Yells)...
49mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems
49mResponding in a Timely Manner
21mOptimizing IT Performance in Complex Systems