rtla timerlat: Debugging Real-Time Linux Scheduling Latency
summary
rtla timerlat is a kernel + userland tool that measures and traces scheduling latency with a few microseconds of overhead (minimum ~2 µs) and can stop and dump an auto-analysis when latency exceeds a threshold (examples: immediate stop at >30 µs, configured 50 µs). timerlat’s kernel timerlat tracer collects minimal Ftrace events, builds histograms of IRQ/RQ/softirq/NMI/thread execution times, decomposes latency into independent parts, detects hardware/hypervisor noise, and points to concrete root causes (example: IRQs disabled by r_spin_lock inside cgroup code).
AI-written summary · about
chapters (17)
- 0:00rtla timerlat introduction
- 1:03Real-time Linux motivations
- 3:30Black-box latency limits
- 5:38timerlat tool architecture
- 7:47Demo of timerlat top
- 11:06Auto-analysis for debugging
- 13:38Real-time scheduling abstractions
- 15:05Latency spike case study
- 19:04Continuous tracing benefits
- 20:51Jitter beyond OS scope
- 22:14Thread latency in non-RT kernel
- 24:44Advanced timerlat tracing
- 27:19Histograms for statistics
- 34:07User space workload analysis
- 36:34Hardware noise measurement
- 38:22Final remarks and future
- 40:13Q&A on hypervisor noise
related talks
51mRTLA Timerlat - the next step in scheduling latency analysis
43mRTLA: Real-time Linux Analysis Toolset
49mDebugging Methodologies for Realtime Issues in Linux Systems
59mProposing a new tracer to monitor RT task behavior
46mMeasuring and Summarizing Latencies using the Trace Event Subsystem
48mCyclictest latency, find out why with tracing
38mLTTng: Beyond Ring-Buffer Based Tracing
37mKernel and userspace tracing with LTTng and friends