The Performance Investigator's Field Guide
summary
After hundreds of field investigations the talk lays out a practical process: get a precise SLA distribution (examples: 95th latency jumping from 60ms to 1,400ms), build a current system diagram, run the USE checklist for utilization/saturation/errors, then drill with stack-collection tools. Shows concrete failures: SMB NetApp latency spikes to ~1,200ms hiding behind misleading "max average over 60s" counters and a CLR GC bug that causes pauses in 250ms multiples; demonstrates real-time ETW work using Event Tracing for Windows and tools like eitrace and live stacks (now on GitHub).
AI-written summary · about
chapters (26)
- 0:00Introduction and approach
- 2:03Structuring investigations
- 5:53Defining the problem
- 13:24Common investigation errors
- 15:27A case study of failure
- 20:53Communication challenges
- 21:46Finding workarounds
- 24:07Looking under the light
- 24:20Understanding your tools
- 29:57The USA method
- 32:37Hardware vs software resources
- 34:15Automating checklists
- 34:41Successful bug diagnosis
- 42:34Tool requirements
- 45:33Tool performance issues
- 50:39Real-time diagnostic tools
- 51:19Stack tracing
- 51:45Visualizing data
- 51:57Summary of tool traits
- 52:35Statistical pitfalls
- 53:52The importance of distribution
- 55:21Visualization necessity
- 56:14Benchmarking limitations
- 57:15Why outliers matter
- 57:57Percentiles in practice
- 59:43Conclusion and post-mortems
related talks
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...
47mResolving performance issues in practice
54mUnderstanding Performance w/DTrace (While The Customer Yells)...
01h 00mSystems @Scale Summer Q&A Week 4
32mSoftware Performance: A Shape Not a Number
47mPractical Performance Analysis
49mDebugging Methodologies for Realtime Issues in Linux Systems
39mResolving Outages Faster with Better Debugging Strategies