Production troubleshooting: boost your skills, a case study
summary
A case-study walkthrough of diagnosing production data corruption during a datacenter migration: Spark jobs produced duplicated and missing rows. The talk explains a methodical troubleshooting approach—forming and testing clear hypotheses, avoiding confirmation bias, eliminating false leads, and distinguishing correlation from root cause—using the Spark incident as an example.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 8:49MapReduce overview
- 12:22Intermittent pipeline failures
- 16:36Debugging cognitive biases
- 22:33Data corruption bug
- 31:38Root cause: floating-point ops
- 38:11Debugging tips & resolution
- 43:10Q&A
related talks
40mProduction Detectives: Solve the Case Before the Crash
41mDetective Hat: Investigating Production Issues
30mOptimize Spark Jobs Using Its UI
46mDiagnosing Production Outages — Why Applications Stop Working
35mProduction as an Experiment Lab
28mResolving Outages Faster With Better Debugging Strategies
39mHow to Use Common Investigation Practices in Troubleshooting
46mProduction Is on Fire? No Stress, I Have strace (and Friends)