Lessons from scaling BPF to detect RDMA Device Drivers Bugs in real time
summary
RDMA tracer uses eBPF fexit probes on a focused subset (≈8 of ~40) of IB control-path functions to capture nonzero return contexts, stores events in a 256 KB ring buffer pulled by a control plane, and emits per‑CPU counters and decoded call stacks. At Meta this identifies RDMA syscall failures that cause ~17% of job restarts (5k+ of 32k GPU hours lost in six months), finds vendor driver bugs, and cuts triage from high tens of minutes to reduced triage time to seconds.
AI-written summary · about
chapters (9)
- 0:00Introduction and motivation
- 3:36RDMA training infrastructure
- 8:17RDMA call errors explained
- 11:31Challenges in tracing RDMA
- 15:24eBPF solution design
- 18:56Data usage and proactive monitoring
- 22:04Production impact and results
- 23:49Vendor driver bug discovery
- 26:21Q&A session
related talks
32mDeveloping eBPF profiler for polyglot cloud-native appli...
25mResilience for AI Workloads at Scale: The Fast and the Finicky!
28mAuto-instrumentation for GPU Performance Using eBPF
29mAuto-instrumentation for GPU performance using eBPF
23mNetwork Communication Debuggability and Observability at Scale
14mAIE-ML dev PMD for VRAN and UPF Datapath Acceleration
21mHigh Network Reliability & Availability in FE & BE for Scalable Training Solutions
47mImprove Load Balancing With Machine Learning Techniques Based on the Sched_ext Framework