How to debug one AI agent run from its observable trace

An agent reports the wrong count, or a reviewer changes an approval but the tool still uses the old value. A final message and one error log rarely show where the failure began. A single-run trace review asks a narrower question: at which observable handoff did this task first depart from its acceptance rule, and is the evidence strong enough to say so?

This page reconstructs one completed or interrupted run. The agent loop guard decides whether the next action is allowed while a run is live. The tool contract defines each interface and executor boundary. The recovery runbook handles ownership and compensation after an external effect. Drift monitoring compares patterns across runs. The artifact here is the event-level evidence record those other decisions can use. For an answer-only quality complaint, the answer failure tree classifies the likely layer; this page checks what each agent step actually did.

Freeze one run and the acceptance gap

Write down the requester’s actual task, the condition for completion, the result shown to the user, and any external state that must be checked. “List every pending quotation” requires an end-of-pagination condition. “Assign according to the reviewer’s edited approval” requires the executed arguments to match that approved version. Without a specific gap, almost every log entry looks relevant.

Locate this execution by trace ID or application request ID. Record its start time and time zone, workflow version, model and prompt versions, tool schema version, and approval policy. One conversation can contain several runs. Do not connect a tool result from an earlier run merely because both share a thread ID. In the OpenAI Agents SDK, a group_id can relate traces in one conversation; it is not a substitute for identifying the particular run. Other frameworks use different names.

Rebuild the event chain from parent relationships

OpenTelemetry describes a trace as the path of an operation and a span as a timed unit of work. Trace IDs and parent span IDs let a reviewer reconstruct a hierarchy across components. Put the root request, model calls, tool execution, subagent handoffs, human approval, and final presentation into a table with their IDs and timestamps. Parallel branches can overlap in time. Adjacent rows in a log viewer are not necessarily causally connected. A tool call ID can join a model’s proposed call to the executor record; an external operation ID can lead to the target system’s actual state.

Keep five observations separate on every row: what action the model visibly proposed; what the executor actually sent after validation and authorization; what the tool returned, including cursors, error codes, and unknown outcomes; which observation the model actually received next; and what the user or human reviewer finally saw. HTTP 200 only means a request returned successfully. It does not prove the task was complete, or that the model received every source. Likewise, a reviewer editing an approval does not prove that the executor used the edited version.

OpenTelemetry’s GenAI conventions describe fields such as execute_tool and a tool call identifier, but those GenAI conventions are still marked Development. This review method does not depend on one attribute name staying fixed. The OpenAI Agents SDK documents generation, function, handoff, and guardrail spans. Check the exact SDK version and exported shape in use, and add application events for approval and final presentation where needed.

Name the first divergence that evidence supports

Start with the acceptance rule and work backward, then verify transitions from the beginning. If the final response omits an item, did the tool supply it? If it did, did the next model input include it and did the model skip a necessary action? If the model proposed the correct action, did the executor change or reject the arguments? Record only observable claims. “The model emitted one list_quotes call” can be checked; “the model privately believed there was only one page” cannot.

Use three conclusion grades. Confirmed means correlated events and authoritative state support the claim. Needs verification means the trace points to a candidate cause, but the tool service or approval ledger must confirm it. Cannot reconstruct means sampling, export delay, missing spans, or corrupted records conceal a decisive step. An absent tool span does not prove the tool never ran. The OpenAI Agents SDK exports batches asynchronously by default, so a dashboard immediately after completion can be incomplete. Check export state, then join application and target-system records. Add instrumentation where necessary rather than treating a blank row as “nothing happened.”

Example one: a quotation list stops after page one

A purchasing assistant is asked to list every supplier quotation still pending this week. list_quotes returns eight rows on the first page and a nonempty next_cursor. The agent then reports “eight in total” without a subsequent call using that cursor. Under the same filters, a reviewer inspects the second page and finds three more quotations.

The first tool call did not fail. The first divergence is the handoff where an available cursor was followed by a claim of completeness. Event rows should preserve the query filters, first-page count and cursor state, the next model-visible observation, the missing continuation, and the final claim. The fix is a completion rule that requires an empty cursor or a stated authorized limit, plus a two-page regression case. If the backend never exported the cursor, the diagnosis remains “needs verification” until that evidence path is repaired.

Example two: an approval edit never reaches the assignment tool

An equipment incident routing agent proposes assigning a ticket to the East on-call team. After checking the site location, a reviewer changes the approved destination to West. The approval record says the West version was approved, yet the executor’s assign_ticket call still carries East. The agent tells the requester that West now owns the ticket. The ticket system identifies East as the current owner.

Align a sanitized summary or fingerprint of the proposed, approved, and executed arguments with the approval version, tool call ID, and ticket operation ID. The first confirmed divergence is that the approved edit was recorded but the execution request did not bind to it. The incorrect final message is a later failed defense. Verify the real owner in the ticket system first; then use the recovery runbook to decide on correction and notice. Do not assign a second time on the strength of the agent’s success message. Regression cases should include edit, reject, expired approval, and arguments changing before execution, not just direct approval.

Make the review sheet useful without collecting secrets

Split the sheet into a run header and repeatable event rows. The header contains the task, acceptance gap, run ID, versions, visibility scope, and reviewer. Each event row records the span and parent, timestamp, visible action, tool call ID, proposed and executed argument summaries, validation and approval outcome, tool result, model-visible observation, external state, and evidence pointer. Finish with the first divergence, alternative explanation, missing instrumentation, smallest safe retest, owner, and closure rule. A second reviewer should be able to reconstruct the conclusion without guessing what a screenshot omitted.

Capture trace content according to the data boundary. OpenTelemetry makes model input/output and tool arguments/results opt-in and warns that they may contain sensitive data. The OpenAI Agents SDK says generation and function spans may capture input/output, with trace_include_sensitive_data enabled by default. Before production use, verify the actual setting, export destination, access controls, and retention. Prefer sanitized IDs, versions, structured status, and controlled evidence pointers. Do not paste customer text, credentials, hidden reasoning, or confidential tool payloads into this site’s workbench: it saves drafts in the current browser. Inspect full source material only in an authorized system. The SDK documentation also says its tracing is unavailable to OpenAI API organizations under Zero Data Retention; use an observation route that meets the local data requirements in that case.

Test one repair against the same case

Preserve an approved test snapshot of the original input, configuration, and available tool responses. State a falsifiable hypothesis about the first divergence: does checking pagination before declaring completion prevent the omission? Does binding execution to the approved argument version prevent stale routing? Replay with synthetic data and change only that one link, checking both the failed case and nearby normal cases. For operations with side effects, check authoritative state first; debugging is no reason to replay a production write.

If the diagnosis stops at a missing event, repair the correlation IDs and boundary events before blaming model behavior. Keep the override reason, first divergence, and closure evidence separate. When similar cases recur across many runs, send the pattern to the drift watchboard to measure its scope.

Boundary

A trace records observable inputs, outputs, and actions. It does not reveal undisclosed internal reasoning, and a successful span does not prove the business outcome is correct. Sampling, asynchronous export, and broken cross-system correlation limit what can be concluded; external writes must be verified in the target system. Single-run review does not replace authorization controls, production alerts, statistical evaluation, or incident recovery. Handle high-impact events under the existing response process and use this sheet to supply reliable evidence.