Autonomous Policy Validation: Building AI Agents to Analyze Logs and Identify...
summary
Built an AI agent pipeline that converts logs into a traversable graph working memory, using hybrid parsing (deterministic regex/schema + LLM) for entity extraction, iterative graph traversal with pruning, and a separate business-justification agent for decisions. System reduces context by 16x, reaches near 100% recall and boosts precision about an order of magnitude, handling >75% of the hardest cases with a three-step traversal heuristic. Trade-offs: token/latency costs, occasional LLM fragility and the need for continuous evaluation.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:19UDAP policy validation challenge
- 3:26AI for log analysis challenges
- 12:29Failed context management
- 14:54AI agent context principles
- 18:27Graph context solution
- 20:15High-level architecture
- 21:16Entity extraction and parsing
- 22:59Graph traversal and enrichment
- 24:46Business justification agent
- 26:10Evaluation methodology
- 28:40Graph approach advantages
- 30:04Implementation lessons
- 32:55System limitations
- 33:16Key takeaways
- 33:58Next steps
related talks
34mAI Agents: Transforming Autonomous Decision-Making
36mAdvanced Evaluation Techniques for AI Agents
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
46mDemystifying AI Agents
38mEnhanced Log Analytics Using Generative AI
25mWho Let the Bots Out? A Guide to Evaluating AI Agents
55mStop AI Agent Hallucinations: 5 Techniques + Production Patterns
24mFrom First Experiments to AI Agents: Adoption Experience and Future Challenges