Tales from an eBPF Program’s Murder Mystery
summary
Cilium’s eBPF ingress programs suddenly disappear, breaking kubelet health checks, because a timing race condition lets another agent install a TC filter that wins handle 1 and Cilium’s atomic swap deletes the unexpected filter. Reproducer uses a looped pod with readiness removed; debug uses bpftrace execsnoop/exitSnoop, a custom netlink capture kernel module and the open‑source ccprobe to trace TC replace/create/destroy events. Short‑term fix raises priorities (default bumped to 10); long‑term fix is upstream BPF links support for TC.
AI-written summary · about
chapters (6)
- 0:00Introduction and context
- 2:00Investigating the incident
- 5:27Understanding traffic control
- 10:00Identifying the root cause
- 14:47Analyzing the race condition
- 19:07Resolution and takeaways
related talks
34mTroubles and Tidbits from Datadogs eBPF journey
42mPanel - Klustered: eBPF Edition
47mUsing eBPF to Bring Kubernetes-Aware Security to the Linux Kernel
40mPanel Discussion: eBPF: A New Era in Cloud Infrastructure Tools
27mLow-Overhead Tracing Using eBPF for Observability into Kubernetes Apps and Services
43meBPF Superpowers for SRE
33meBPF for Security
20mDebugging Kubernetes and Container Workloads with the Superpower of eBPF