Advanced Linux Kernel Networking Monitoring
summary
Built a kernel-level monitor called net-exporter that uses eBPF tracepoints (kfree_skb, napi/softirq entry/exit) to stream labeled events and stack traces into a Go agent, map sizes ~10k and per-CPU arrays, and push metrics to Prometheus. It measures packet drops and softirq scheduling vs execution latency by timestamp differences across three tracepoints, finds root causes (e.g., a buggy kernel module starving softirqs), runs with low overhead by filtering in-kernel and sending only outliers, and proved effective in production.
AI-written summary · about
chapters (17)
- 0:00Intro
- 2:03Container networking
- 2:41Kubernetes networking semantics
- 6:48An incident - Triage
- 9:33An incident - Conclusion
- 11:09Existing monitoring methods
- 13:31Ideal monitoring method
- 14:05Network stack
- 17:10Receiving data what may go wrong
- 20:37Sending data what may go wrong
- 21:44Targets and Methods
- 25:11Monitoring packet droppings
- 32:36Monitoring packet latencies
- 39:46Metrics and Events
- 40:33Putting all together
- 41:56Runtime Optimizations
- 43:00Compiling Optimizations
related talks
31mZero instrumentation observability with eBPF
20mNetwork Monitor: A Tale of ACKnowledging an Observability Gap
38mLinux Kernel Tracing Using eBPF
27mLow-Overhead Tracing Using eBPF for Observability into Kubernetes Apps and Services
38mThe Next Linux Superpower: eBPF Primer
42mCommunity Driven Breakout Session: eBPF and Auto-Instrumentation
42mKernel Analysis Using eBPF
32mTracing the Linux networking stack with Retis