You Can't Fix a Prompt You Can't See Fail
summary
Shows a concrete fix: wrap the planning step in an OpenTelemetry plan span that stores the list of tools, then run a separate answer check and a plan check. Demo sends spans to Jaeger demo and runs the release-helper five times (scripted) to show one case where the answer passes but the plan reads yesterday’s summary. A benchmark, the agent telemetry benchmark, finds standard logs catch 6 of 14 fault types while adding planning spans catches all 14. Recording the plan takes only a few lines and reliably tells whether to change prompt or fix a tool.
AI-written summary · about
