Network Monitor: A Tale of ACKnowledging an Observability Gap
Talkat SREcon19 Europe/Middle East/Africa · on Wednesday, 2019-10-02· at 15:00· Track 1: The Liffey B·
summary
A Redis connectivity incident during peak sales triggers six weeks debugging and exposes an observability blind spot after engineers spot high TCP retransmits that force cloud provider intervention. The response is a lightweight eBPF daemon on every node that attaches kprobes to TCP connect/finish, records microsecond connection timings in BPF maps, exports tagged metrics to Prometheus and Datadog, and delivers reduced timing noise vs userspace so teams can alert and localize network faults without noisy user‑space jitter.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:28Jason Shopify
- 1:47Open Circuit Breakers
- 2:25Black Friday Cyber Monday
- 4:28TCP Components
- 6:57Network congestion
- 7:12Shark Bite
- 7:24Investigating the Problem
- 8:00Ping Mesh
- 9:34User Space
- 10:19EBPF
- 12:47BPF Program
- 15:29IOvisor
- 17:05Multi Cluster Setup
- 18:04Data Dog
- 18:29Observation Gap Fill
- 19:00Summary
related talks
37mMonitoring Service Architecture and Health with BPF
18mMethodology and Practice in Observing Kernel Networking
46mLessons learned from monitoring a large production network
31mZero instrumentation observability with eBPF
35mMonitoring Microservices
39mNetwork Observability with Envoy
44mAdvanced Linux Kernel Networking Monitoring
32mActive Fault Finding in Networks