Advanced Tracing Features to See What Your Application is Waiting On
summary
New kernel features plus two new libraries let users find what an app waits on by actually measuring and matching events: libtracefs (and a tracing client Trace-command) creates synthetic events that match sched_waking to sched_switch, records per-PID deltas, and builds histograms with ~2000 buckets inside the kernel. Example tooling extracts wake‑latency histograms, prints kernel stacks for long uninterruptible sleeps, and uses S-frames user-space unwinding; prototype works (6.3+ kernel, tested locally) though a few fixes are pending.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 0:46Ftrace infrastructure
- 5:00Libtracefs and Trace-cmd
- 10:26Analyzing trace data
- 17:36Data processing with Trace-eval
- 23:31Synthetic events
- 32:46Histograms in the kernel
- 36:46Advanced debugging techniques
- 44:25Q&A and future directions
related talks
41mAnalyzing Your System with Tracing Libraries
48mCyclictest latency, find out why with tracing
35mLinux Tracing updates
49mAdding Inter-event Capabilities to Linux Kernel Trace Events
18mScheduler Tracing With ftrace + eBPF
55mRecent Advances in Linux Tracing
41mSynthetic Events and Tracing Latency Within the Kernel
31mFantastic tracepoints and where to find them