An AI agent can produce a flawless-looking trace and still leave the business in the wrong state. That is the evaluation gap enterprise architects should worry about.
A common answer to agent reliability is more observability: retain prompts, tool calls, intermediate reasoning, approvals and final messages. Those records are necessary, but they do not prove that the intended transaction happened once, under valid authority, with the correct side effects. A trace describes what the agent attempted. The business outcome lives in the systems the agent changed.
The practical thesis is simple: an enterprise agent should not pass an evaluation until its claimed result reconciles with authoritative system state and the applicable business invariant.
The requirement matters most when agents can create consequential or difficult-to-reverse state changes; a research agent drafting a summary does not need the same evaluation architecture as one issuing payments.
A refund exposes the difference
Imagine a customer-service agent handling a €750 refund. It reads the order, checks policy, requests approval, calls the payment tool and tells the customer the refund is complete. Its trace is orderly. Every tool call has a timestamp. The approval appears in the transcript. The final answer is polite and accurate in form.
But the payment API timed out after accepting the instruction. The agent retries. One call settles immediately; the other clears later. The customer receives €1,500.
A transcript grader may reward the agent for following the expected steps. A production-grade evaluation asks different questions: did exactly one refund exist in the payment ledger, did it reference the correct order, was the amount within the approver’s authority, did the customer record reflect the same status, and could the system safely compensate for partial success?
Why evaluation changes when agents can act
Evaluating an assistant is largely about the quality of its output. Evaluating an agent is harder because the output may be only a description of changes already made elsewhere. Once an agent can issue refunds, update customer records, modify access rights or deploy software, the final message is no longer the result. The changed environment is.
Anthropic makes this distinction explicit in its evaluation guidance: a trace records what happened during an agent’s execution, while the outcome describes the resulting state of the environment. An agent can therefore produce a plausible trace and report success while the underlying task has failed.
This changes what an enterprise evaluation architecture needs to measure. The question is no longer only whether the agent selected reasonable tools and followed an acceptable process. It is whether the resulting business state is correct. That requires ground truth outside the agent itself: transaction records, system state, authorization decisions and the business rules that determine whether the outcome is valid.
The architectural consequence is significant. As agents gain authority to change production systems, evaluation moves from judging AI behavior toward verifying transactions and business outcomes.
Observability is evidence, not a verdict
A complete trace is invaluable for diagnosis and audit. It can reveal which tools were invoked, what the agent observed and where an error began. Yet observability cannot turn an invalid action into a valid one. A beautifully logged duplicate payment is still a duplicate payment.
This is the non-obvious architectural consequence: the evaluation layer needs read access to authoritative business state, not just the agent platform’s telemetry. It must reconcile across boundaries the agent itself may not own. For a refund, that could mean the order system, payment ledger, approval service and customer case. For a software deployment, it could mean the change record, artifact digest, production version, health checks and rollback state.
That requirement also prevents the agent from grading its own homework. The component proposing or executing an action should not be the sole authority declaring that action successful. An independent evaluator should obtain ground truth from systems of record and apply deterministic checks wherever the business rule is deterministic.
There is an important boundary here. The agent evaluation layer should not recreate transactional integrity. Systems that execute transactions must continue to enforce their own invariants.. Evaluation verifies that the combination of agent, tools, workflows and underlying systems produced the intended business outcome.
What “independent” means
An independent evaluator does not need to be another AI agent or a separate platform. Independence means that the component executing an action is not the sole source of truth for whether that action succeeded.
The evaluator should retrieve authoritative state independently from the systems affected by the transaction and apply business rules that the executing agent cannot redefine. Where an outcome can be verified deterministically—such as a ledger balance, authorization decision, deployment version or transaction status—the evaluator should prefer that evidence over another model’s interpretation.
For the refund example, the agent may report that €750 was refunded, but the evaluator verifies the payment ledger, order state and approval record directly. The agent’s trace remains evidence of how the action was attempted; it is not proof that the business outcome is correct.
Architecturally, the separation is therefore between execution and verification, not necessarily between two AI systems.
Evaluate the invariant, not the preferred path
Teams often encode one expected sequence and mark deviations as failures. That can overfit the evaluation to today’s orchestration. A capable agent may find a legitimate route the test designer did not anticipate. Conversely, an agent may follow the expected route while producing a prohibited result.
The durable test is therefore the business invariant. In the refund scenario:
- the refund corresponds to an eligible order;
- the total refunded amount does not exceed the authorized amount;
- the acting identity has valid authority at execution time;
- retries cannot create duplicate financial effects;
- all dependent records converge to a consistent state;
- a failed or partial transaction has a tested recovery path.
NIST’s 2026 work on software-agent identity and authorization reinforces one part of this design: agents with access to data, tools and applications require appropriate identification and authorization controls. Evaluation should verify those controls at the moment of action, rather than infer authority from a prompt or a historical approval embedded in the trace.
The cost is real—and worth making visible
Outcome-based evaluation is more expensive than grading text. It requires test environments with meaningful state, adapters to systems of record, synthetic or carefully governed data, replayable failure conditions and agreement between technology and business owners about invariants. Some outcomes arrive late, making immediate pass/fail judgments impossible. Independent reconciliation also adds latency and operational complexity.
That cost should not be hidden behind a generic “AI observability” budget. It is part of automating the transaction safely. If an organization cannot identify the authoritative state or express the invariant, the process is not ready for autonomous execution—regardless of the model’s benchmark score.
A test enterprise architects can require
Before an agent receives production write access, require an evaluation suite proportionate to the consequence and reversibility of the actions it can perform. Higher-risk actions should require broader failure coverage, repeated trials and stronger evidence of safe recovery before deployment.
At minimum, the evaluation should cover normal execution and the failure conditions relevant to the process: timeouts, retries, duplicate responses, expired authorization, rejected approvals and partial downstream failure. Critical scenarios should be repeated where model or environmental nondeterminism could change the outcome.
For every trial, require an independent evaluator to reconstruct:
- the initiating identity and delegated authority;
- the intended business change;
- the tools and policy decisions used;
- the final state in every authoritative system;
- any compensation or recovery action.
The acceptance criterion should follow business risk rather than an arbitrary number of successful trials. Low-impact, reversible actions may tolerate occasional recoverable failures. Financial, security-sensitive, regulated or difficult-to-reverse actions should require substantially stronger evidence, including zero unauthorized actions and demonstrated recovery from every critical failure mode tested.
The principle remains falsifiable: the agent passes only when its observed outcomes meet the acceptance criteria defined for that class of business action. A clean trace without a correct outcome is a failed trial.
The goal is not to prove that the agent behaved plausibly. It is to prove that the enterprise remained correct after the agent acted.