Active Fault Finding in Networks
summary
Describes building active probes at Facebook scale: open-sourced fbtracer to enumerate ECMP paths, a UDP pinger/responder system called net-norad with ~1:100 pinger:responder ratio and up to ~1,000,000 packets/sec from a host, and a Network Fault Isolation (NFI) system that sends thousands of TCP/thrift source-port–controlled requests per second. Detection latency is ~20s after 2–3 minute metric ingestion; C++ rewrites cut probe latency from ~6–7ms to ~1ms. The pipeline locates a misbehaving ASIC (hidden counters) and successfully isolates links among thousands of ECMP paths.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 1:02The First SRE Principle
- 2:01Data Center Networks
- 4:11Traceroute
- 5:23Backbone
- 6:22SRE Principle
- 7:32NetNorad
- 7:56NetNorad evolution
- 10:34Network Hierarchy
- 12:53Alarm
- 13:34Isolation
- 14:59Alarming
- 15:16Network Fault Isolation
- 17:33Thrift Requests
- 18:41Loss
- 20:43Results
- 21:10The discovery
- 22:49Issues
- 23:51Restructure
- 25:08C
- 27:12Whats next
related talks
29mNetwork Fault Finding System: Packet Loss Triangulation
46mLessons learned from monitoring a large production network
20mNetwork Monitor: A Tale of ACKnowledging an Observability Gap
30mNoisy Neighbors, through Networking
26mF10: A Fault-Tolerant Engineered Network
44mNetsec is Dead(?): Modern Network Fingerprinting for...
23mNetBouncer: Active Device and Link Failure Localization in Data Center Networks
46mSelf-Programming Networks: Network Clock Synchronization, Event Playback and Real-time Action