007: Democratically Finding the Cause of Packet Drops
summary
007 is a host-only deployment that diagnoses per-TCP packet loss by subscribing to retransmission events, issuing one traceroute per retransmitting flow, and letting each affected connection cast a per-connection voting on link blame; an integer voting rule then ranks culprit links. During a two-month Azure deployment it finds every problem prior monitors did and uncovers others (software bugs, FCS errors, route flaps). The algorithm has provable accuracy (100% under a balance bound), shows perfect detection down to about 0.05% drop rates, yields ~10% false positives when the bound is violated, and tolerates traffic skew up to ~50% before accuracy degrades.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:41Not all faults are the same
- 1:50Mapping complaints to faulty links
- 2:17Clouds operate at massive scales
- 2:32Low congestion drop rates add noise One-off packet
- 3:22Talk outline
- 3:40Solution Requirements
- 3:54Want to avoid infrastructure changes
- 4:19A "strawman" solution
- 4:44Example of doing tomography
- 5:55Tomography is not always practical
- 8:12We can fix problems with traceroute
- 9:08How the system works
- 10:27Can diagnose TCP flows
- 11:04Attractive features of 007
- 11:15We give theoretical guarantees
- 11:50Questions to answer in evaluation
- 12:17Does 007 work in practice
- 13:05Are votes correlated with drops?
- 14:09Comparison to MAX COVERAGE
- 14:51Binary optimization underperforms
- 15:35Is 007 robust to noise?
- 15:54Skewed traffic causes problems
- 16:27Is 007 impacted by traffic skew?
- 17:04Conclusion
related talks
29mNetwork Fault Finding System: Packet Loss Triangulation
32mActive Fault Finding in Networks
35mWho Dropped the Packet? Solving K8s Network Mysteries in Real-Time
20mNetwork Monitor: A Tale of ACKnowledging an Observability Gap
16mFrom Source to Solution: Tackling Packet Losses in Large-scale Cloud Gaming...
23mNetBouncer: Active Device and Link Failure Localization in Data Center Networks
33mWhose Packet Is It Anyway? Life of a Packet Through a Service Mesh
53mWhat is eating my traffic? Packet thieves revealed