AI flame graphs with eBPF
summary
An open source AI flame graphs tool is built that merges eBPF context with Intel EU stall hardware sampling to produce unified CPU+GPU flame graphs showing GPU instruction frames (typically 2–6 deep) up to the exact instruction offset and stall reason, plus the CPU stacks that launched them. The prototype uses Intel eustall hardware profiler and runtime parsing (Neo), faces a Kprobe vs Uprobe tradeoff (Kprobes simpler but high overhead, uprobes lower), aims for target <0.1% overhead, and is available on GitHub and runnable on Intel consumer GPUs.
AI-written summary · about
chapters (12)
- 0:00Introduction to AI flame graphs
- 3:11Intel AI profiler overview
- 8:34Correlating CPU and GPU data
- 12:09Challenges and caveats
- 15:20Flame scopes and visualization
- 17:03Provis CLI tool demo
- 19:11eBPF's role in development
- 20:32Future of AI profiling tools
- 23:41Q&A: Synchronous execution
- 25:08Q&A: Multi-kernel mapping
- 27:45Q&A: Comparison with GPU traces
- 29:30Q&A: Overhead details
related talks
29mAuto-instrumentation for GPU performance using eBPF
32mDeveloping eBPF profiler for polyglot cloud-native appli...
28mAuto-instrumentation for GPU Performance Using eBPF
14mKPerfIR: Towards a Open and Compiler-centric Ecosystem for GPU Kernel Performance...
28mArray Programming on GPUs: Challenges and Opportunities
12mContinuous Profiling for GPUs
10mGraph Based Pipeline Parallelism
25mOptimizing LLM Efficiency One Trace at a Time on Kubernetes