Medic for Apache Spark - First Aid for Failing Jobs
summary
A diagnostics agent called Medic for Apache Spark automatically ingests logs and metadata and returns evidence-backed root causes and fixes in minutes rather than hours, with concrete reductions in false leads and token use. Key components include end-to-end test harness with record/playback fixtures and offline evals (eg a three-fix limit), exception classifier that fingerprints clusters and ranks exceptions, metrics quarantine sub-agent that collages annotated graphs into images, and a multi-agent LangGraph architecture replacing a single-prompt React agent that improves reliability and extensibility.
AI-written summary · about
chapters (10)
- 0:00Medic for Apache Spark intro
- 1:26Vision for automated diagnostics
- 2:53Prototype challenges
- 3:58Improving observability
- 5:22Log exception classification
- 6:45Metrics quarantine analysis
- 8:09Multi-agent architecture
- 9:14Spark SQL optimization
- 10:18Current architecture plans
- 11:02Closing and contributors
related talks
13mLogging Apache Spark How we made it easy
32mScalable Monitoring Using Prometheus with Apache Spark
41mOptimising Apache Spark and SQL for Improved Performance
44mRiak, Redis, Apache Solr & Spark: Deploying with the Basho Data Platform
13mBeyond Manual Firefighting
30mAI Agents for Incident Management
35mAgentic Incident Response
40mScaling with Apache Spark (or a lesson in unintended consequences)