Finding the Needle in a Haystack: Tenant-Level Network...
summary
Cloud-scale tool traces tenant flows end-to-end by deploying agents on bare metal and virtual switches, mirroring headers and collecting data in a central collector that correlates ingress/egress to expose the failed hop. It combines eBPF/XDP tracing, sampled sFlow/netflow and controller-driven mirroring, explores packet coloring (VXL header) and in-band telemetry, and handles privacy truncation and sampling. Production deployment on millions of nodes shows 300+ users, ~2k investigations/month and up to an 8% troubleshooting speedup, with safeguards, hard limits and one-click stop controls.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 0:41Packet flow in cloud networks
- 3:07Virtual private cloud and encapsulation
- 5:14Network failures impact on tenants
- 7:18Telemetry and monitoring in cloud networks
- 9:04Limitations of traditional monitoring tools
- 11:33Tenant-level troubleshooting architecture
- 12:55Traffic mirroring from virtual switches
- 14:17Monitoring virtual devices with flexible configs
- 15:42Tracing user-space apps with eBPF
- 17:30Monitoring XDP programs in kernel
- 18:12Tracking AF_XDP apps for visibility
- 19:57XDP offload and hardware acceleration challenges
- 20:38DPDK monitoring limitations
- 21:22Exploring ENPU for hardware networking
- 22:04Monitoring physical devices with packet marking
- 23:47VXLAN header for packet tracking
- 24:50In-band telemetry challenges
- 25:33Mapping customer networks to layers
- 26:35Lessons learned and best practices
- 28:19Future directions and closing
related talks
15mBuilding a Network Telemetry Platform
32mDistributed Tracing in Action: Our Journey with OpenTelemetry
01h 00mSystems @Scale Summer Q&A Week 4
50mDistributed Tracing in the Wild
51mAdvanced approaches to traffic inspection & network diagnosis w/
39mFinding the Needles in a Haystack: Identifying Suspicious Behaviors...
49mRelational Observability for Cloud-Native Security and Data Science
49mVisibility Without Borders: NETSCOUT to help migrate apps to