Your Agent Failed in Prod. Good Luck Reproducing It.
summary
An architecture for reproducible postmortems that records every model call, tool payload, memory read and state transition into an append-only event log and replays failed runs deterministically. Claims include turning an unreproducible $190,000 accidental trade into a line-by-line trace, stubbing nodes to run targeted live tests, and asserting tool-level guardrails without calling models. Implementation details: Chronicle captures inputs/outputs and metadata; the Boundary annotation marks nodes for recording and selective replay, enabling deterministic CI for debugging and regression testing.
AI-written summary · about
chapters (11)
- 0:00Agent failures in production
- 1:03Why production bugs are hard to reproduce
- 2:49Temperature zero and agent issues
- 3:33First principles of non-reproducibility
- 5:21Bitwise determinism vs replayability
- 6:46Recording agent runs at boundary
- 7:27Chronicle for replayability
- 8:09Demo: recording agent traces
- 10:11Testing guardrails with traces
- 12:19Deterministic vs behavioral testing
- 13:21Key takeaways for debugging
related talks
01h 20mMind the Gap (In your Agent Observability)
42mReplay Production Traffic Locally: Instantly Spot & Fix Breaking Changes
26mBeyond Sandboxes: Architecting Durable Runtimes for AI Agents
30mArchitecting Integrity: Responsible Software Evolution in the AI Era
44mAgentic Observability — The Path to AI Co-SRE
32mWe're sorry, but something went wrong
25mLLM Agents for Site Reliability
12mDay 2: From Foundations to Production