All Your Queues Are Belong to Us: The Hunt for a Network Bug...
summary
Datadog finds a kernel bug that forces all egress traffic onto a single transmit queue, causing 16 Gbps pod‑to‑host throughput instead of the instance’s 25 Gbps, repeated packet drops (~2%) and elevated latency (~10 ms). Tracing with bpftrace reveals an SKB q_mapping set across network namespace boundary introduced in a 5.11.11 commit (backported to Ubuntu 24.2+). Fix is a small kernel patch plus an eBPF mitigation (via Cilium) that resets q_mapping and restores parallel queues and native performance.
AI-written summary · about
chapters (20)
- 0:00Network bug hunt intro
- 1:45Kafka workloads performance drop
- 3:31Network issue symptoms
- 4:15Single transmit queue impact
- 6:26fq_codel and queue disciplines
- 10:00Pod egress traffic imbalance
- 12:36Manual namespace testing
- 13:38Mitigation with fq instead of fq_codel
- 15:21Affected systems and versions
- 16:24Kernel transmit queue debugging
- 19:10SKB and queue mapping fields
- 21:15BPF trace for queue mapping
- 23:21Packet transmission tracing
- 25:24Kernel change causing queue issue
- 26:50XPS role on GCP
- 28:14Root cause analysis summary
- 29:17Kernel patch and BPF mitigation
- 31:21Fixes results
- 32:43Business metrics impact
- 33:28Conclusion and lessons
related talks
27mDebugging Linux Issues with eBPF
07mLightning Talk: Taming Noisy Network Neigbours with Cilium and eBPF
30mNoisy Neighbors, through Networking
34mIt's Not Always the Network (But Here's How to Prove It): Kubernetes Packet...
22mLIVE Q&A #3
31mBeware of the kernel RTNL mutex
34mTroubleshooting Low Latency Application on CNF Deployment
29m“Are These Things Talking to Each Other?” - Observing Kubernetes Networking...