Production is Priority - Self Fix / Heal Techniques
summary
Describes a concrete remediation framework that ties Nagios alerts to an application server (Flask + MySQL) which pushes commands to node clients over ZeroMQ, runs scripts against Hadoop components (HDFS, tasktracker, namenode), records recorded stdout and exit codes, and retries checks or chains followups like disk, network, or jobtracker probes. Includes examples of parallel fixes across hundreds of hosts, integration notes about Ganglia and Monit, and a beta open source client for automated recovery called out.
AI-written summary · about
chapters (6)
- 0:00Start
- 4:29Common Monitoring Tools Used
- 5:24General Process
- 6:45Our Approach
- 10:56Advantages
- 12:26Screen Shots
related talks
30mA Well-Maintained Production Application
39mHow to Use Common Investigation Practices in Troubleshooting
36mIt's not in Production Unless it's Monitored
31mAlerting for Distributed Systems—A Tale of Symptoms and Causes, Signals and Noise
40mProduction Detectives: Solve the Case Before the Crash
29mProduction Cluster Monitoring and Remediation for High Reliability
28mDiagnostic Tooling - Be Ready when Things Go Wrong in Production
24mListen to Production the Way It Deserves