Tracking Down What Happened: A Comparison of Tracing Solutions
summary
Compares four Linux tracers — ftrace, eBPF (bpftrace), SystemTap and LTTng — by tracing IPC in hackbench (20 senders × 20 receivers = 400 IPC events) on Fedora 34; measures every send→receive latency and captures sched_wakeup/sched_switch around the single worst transfer (5.6 ms worst-case reported by eBPF). eBPF has the least runtime impact (+31%) and immediate in-kernel histograms (ebpf fastest); ftrace adds +44% and needs manual parsing; SystemTap is slowest (+84%); LTTng requires adding USDT points — had to patch hackbench with USDT.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 2:35Interprocess communication
- 4:20Conditions for testing
- 5:54Application for testing
- 8:12What are trace points
- 15:54FTrace
- 17:55FTrace Demo
- 20:47Python Histogram
- 21:34Kernel Shark
- 26:00EPPF
- 28:44BPF Trace
- 29:39EPF Trace
- 30:40Schedule Switchings
- 31:54Summary
- 33:24SystemTap
- 36:07SystemTap Script
- 36:47Histogram
- 38:12System Tab Advantages
- 41:23LTTG Advantages
- 44:08Post Processing
- 45:16Python Post Processing
- 46:05Tracing Compass
- 47:11Lttng
- 51:09Final Thoughts
related talks
37mKernel and userspace tracing with LTTng and friends
42mHow Much Is Tracing? Measuring the Overhead Caused by the Tracing Infrastructure
46mbpftrace internals
34mTracing: The Bane of You Security Folks
25mKernel tracing with BPF
55mRecent Advances in Linux Tracing
38mLinux Kernel Tracing Using eBPF
38mLTTng: Beyond Ring-Buffer Based Tracing