Measuring and Summarizing Latencies using the Trace Event Subsystem
summary
RT-devel merges an inter-event patchset (version 9) that builds on the Trace Event subsystem to compute latencies between two events, generate per-key histograms, and emit synthetic events for further aggregation. It supports on-max handlers, snapshots, and a tag/delete mechanism to avoid histogram-table growth when keys are object addresses (skb/request). A helper script func_event_latency auto-creates function tracepoints and histograms; examples show sched, networking (skb), and block request latencies producing working histograms and max-latency snapshots, code in the linked git repo.
AI-written summary · about
chapters (18)
- 0:00Intro
- 3:19Trace Events Background (cont'd)
- 8:19Latency is a derived quantity
- 9:59Latency (cont'd)
- 11:09We have a latency value, now what?
- 12:19Handlers and Actions (onmax and save)
- 13:06onmax and save output
- 14:06Chaining handlers (add snapshot())
- 15:19onmax and save and snapshot output
- 16:05What about a latency histogram?
- 19:52onmatch synth and save/snap output
- 28:09Object latencies (using delete())
- 29:14Object latencies, part 2
- 33:57Function Event Magic
- 36:04Function Event Latency
- 37:57Generate latency hist for any 2 events
- 39:04General purpose latency script output
- 41:35Questions?
related talks
49mAdding Inter-event Capabilities to Linux Kernel Trace Events
41mSynthetic Events and Tracing Latency Within the Kernel
50mrtla timerlat: Debugging Real-Time Linux Scheduling Latency
51mRTLA Timerlat - the next step in scheduling latency analysis
35mTracing is More Than Traces: The Insights in Trace Aggregates
59mProposing a new tracer to monitor RT task behavior
38mLTTng: Beyond Ring-Buffer Based Tracing
48mCyclictest latency, find out why with tracing