Detective Hat: Investigating Production Issues
summary
A pragmatic checklist for chasing production bugs: five core laws—all production issues are failures of process, believe the problem until you prove it isn’t, write a live detective's notebook with problem, impact, potential causes, complications, unanswered questions and tomorrow’s problems. Use tools like Obsidian or Google Docs, Slack for status, Mixpanel and Twilio logs, SQL queries (example: prove it happened 68 times), add in-situ logs, reproduce step-by-step, push one change or toggle a feature flag, communicate updates frequently, and run a postmortem to learn.
AI-written summary · about
chapters (9)
- 0:00Detective Hat approach
- 3:55Production issues and impact
- 8:29From development to production
- 11:20Detective's notebook
- 19:48Modeling incidents and doubts
- 25:25Reproducing problems
- 30:22Narrowing down issues
- 33:17Incident communication
- 37:54Learning from production
related talks
40mProduction Detectives: Solve the Case Before the Crash
26mA Practical Taxonomy of Bugs
26mow to Enjoy Debugging in Production with Karen Tracey
39mHow to Use Common Investigation Practices in Troubleshooting
26mThe Mindset of Debugging
12mDebugging Python in Production: Practical Techniques Beyond Print Statements
25mHow to detangle and understand production
49mObservability in Practice: How to Become Proactive with Bugs