Evolvable Network Telemetry at Facebook
Talkat 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI '22) · 2022-04-04..04-06·
summary
Production telemetry system PCAT introduces a uniform change cube representation, an intent-based layering that splits intent, data, collection and job models, and tooling to attribute and explore change dependencies. Facebook measures up to 30 commits and ~1,000 live code changes per week per component; two real incidents (vendor changing a linecard version from int to string that hid circuits, and averaging 90th-percentile sub‑switch CPU values that missed alerts and caused packet drops) motivate the design. Real-time topology derivation using cube-triggered jobs cuts derivation delay by over 100 seconds on average; PCAT runs in production.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:27Network Telemetry is Critical for Network Managem
- 1:11Key Challenge for Telemetry in Production: Evolvability Network devices and management applications are constantly evolving
- 2:08Magnitude of Changes
- 2:42Incident 1: Changes Affect Many Components
- 4:07Incident 2: Data Misinterpretation
- 5:22Bringing Changes to First-Class Citizens in Telemetry
- 5:45PCAT: Production Change-Aware Telemetry System
- 6:34Change Abstraction: Change Cube
- 8:35Change Attribution: Three Generations of Telemetry
- 9:08and Gen2
- 10:22's Problems
- 11:08PCAT) Layering Design
- 13:11Change Exploration: Topology Derivation Creates derived topology from normalized device-level data.
- 14:12Open Questions: Adaptive Telemetry Primitives
- 14:50Open Questions: Trustful Telemetry
- 15:40Summary
related talks
46mLessons learned from monitoring a large production network
21mLive Traffic Load-Testing-Measuring and Validating Capacity at Facebook
15mBuilding a Network Telemetry Platform
19mEngineering Facebook's Data Center Network: Challenges and Limitations
32mActive Fault Finding in Networks
38mFacebook's Data Center Fabric
26mEvolution of Application Visibility for Telcos: From Programmable Switches to Cloud Native
38m2022 Observability TAG Update