Lessons learned from monitoring a large production network
summary
Facebook monitors a multi‑datacenter, multi‑continent network with two backbones (classic MPLS and software‑defined Express) using four monitoring tools that combine SNMP counters, ICMP/UDP/TCP synthetic probes and a novel eBPF‑based TTL tagging system. Agents send millions of probes to far fewer pingers, aggregators compute loss and RTT, and end‑to‑end injected faults plus synthetic aggregation tests measure coverage over time and space. Small team ownership (≈1.5 engineers per tool) forces simplicity; tools largely work but regressions, coverage holes and dependency cascades drive automated end‑to‑end tests and SLIs.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:56How do we monitor the network
- 1:33Fabric networks
- 7:20Tools
- 12:00Localisation
- 16:00Examples
- 17:37Multiple tools
- 24:40Coverage
- 28:15Measuring
- 34:20Handling failure
- 37:35Keeping up with scale
- 39:15The tools
- 41:45Questions
related talks
32mActive Fault Finding in Networks
20mNetwork Monitor: A Tale of ACKnowledging an Observability Gap
16mEvolvable Network Telemetry at Facebook
38mFacebook's Data Center Fabric
21mHigh Network Reliability & Availability in FE & BE for Scalable Training Solutions
19mEngineering Facebook's Data Center Network: Challenges and Limitations
14mSkyline: A Cloud Centric Internet Monitoring Engine
29mFinding the Needle in a Haystack: Tenant-Level Network...