The Holy Grail of Systems Analysis: from What to Where to Why
summary
Claims that distributed latency usually comes from throughput-driven contention and queueing, illustrated with Google Dapper’s limits and a demo built on LightStep and open tracing. Shows a small code change that tags spans with resource identifiers and then aggregates contending spans to expose causation not just correlation, yielding surprising numbers like 70–80% of lock load from a different endpoint and single actors causing ~88% of contention, and a working demo with mutex tracking.
AI-written summary · about
chapters (27)
- 0:00Introduction
- 1:46Building systems is hard
- 2:31The purpose of monitoring
- 3:08Microservices
- 4:08Microservices and tracing
- 5:35Distributed systems
- 7:24Metrics
- 8:14Anomalies with Metrics
- 10:03Dashboards
- 11:19Number of Metrics
- 12:20Where is the Problem
- 13:40Our Architecture
- 14:04Queues
- 14:49Donuts
- 18:30Chasing
- 19:42Sampling
- 21:03Understanding Why
- 21:26Hypothesis
- 22:06Independent Tracing
- 23:44Preview
- 24:51Back to doughnuts
- 25:02The doughnut zone
- 26:27Graph layout library
- 28:01Restock
- 30:29Cleaning the fryer
- 32:48Summary
- 33:40Open Tracing
related talks
35mTracing is More Than Traces: The Insights in Trace Aggregates
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...
46mHow to Properly Blame Things for Causing Latency
57mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems
41mTracing Real-Time Distributed Systems
01h 01mThe Performance Investigator's Field Guide
25mProfiling Request Latency with Critical Path Analysis
34mReduce Complexity with Distributing Tracing