Why Agentic Systems Need Closed-Loop Evaluation

Agentic systems need closed-loop evaluation because a good final answer is not enough evidence to authorize a real effect. A trustworthy release separates hard safety invariants, final business state, execution trajectory, retrieval quality, repeated-run reliability, operational cost, and causal diagnosis. It then converts confirmed failures into regression cases before the next deployment.

That is the central claim. The smallest complete model has three boxes:

flowchart LR
  run["Agent run"] --> evidence["Independent evidence"]
  evidence --> decision["Release decision"]
  decision -->|"confirmed failure becomes regression"| evidence

The loop matters more than any individual metric. Without it, evaluation becomes a report written after the system changes. With it, evidence governs what may change next.

A successful sentence can hide a failed system

Consider an enterprise access agent asked to restore Sarah’s Salesforce access. It searches policy, finds the employee, inspects permissions, proposes a change, requests approval, applies the change, and verifies final state.

“Sarah’s access has been restored” can still be wrong in several distinct ways:

• it found Globex Sarah while the authenticated tenant was Acme;
• it applied a permission not supported by current policy;
• it mutated state before approval;
• it received an adapter success response but never verified authoritative state;
• it succeeded once but fails four of five identical trials;
• the system worked, but a broken evaluator marked it wrong.

These are not variations of one quality number. They have different owners, remedies, and release consequences.

Anthropic’s agent evaluation guidance recommends starting from real failures, combining deterministic and model-based graders, and treating evaluation as an iterative discipline. The practical consequence is architectural: evidence collection and grading cannot live only inside the agent being judged.

Keep hard gates outside the average

A useful result contract keeps dimensions separate:

Never average a cross-tenant disclosure with a helpful explanation. A single critical failure rejects the candidate, even if every quality average improves.

Evaluate final state and trajectory independently

Outcome and trajectory answer different questions. Final state tells us whether the business result happened. Trajectory tells us whether necessary constraints held along the way.

Suppose two agents restore the correct permission. One reads policy, proposes an effect, obtains approval, applies it, and verifies state. The other writes first and asks for approval later. Outcome-only grading passes both; trajectory evidence correctly rejects the second.

The reverse matters too. An agent may take an extra read-only inspection and still be safe and correct. An exact reference-trace matcher would fail it unnecessarily. Good trajectory graders protect invariants—tenant-safe lookup, approval before effect, matching proposal digest, bounded retries, final verification—without demanding one brittle path.

Measure retrieval before blaming generation

Retrieval-augmented agents contain at least two evaluable systems. Retrieval selects evidence; generation interprets it. If the decisive policy passage never reached the model, prompt tuning cannot repair recall. If it arrived and the model contradicted it, changing the retriever obscures the cause.

Record immutable retrieved IDs separately from generated citations and effects. Then report recall@k, context precision, groundedness, citation correctness, and tenant-leak rate separately. The RAGAS paper and metric catalogue offer useful metric definitions, but executable tenant and state assertions remain release authority.

Authorization also precedes ranking. Tenant scope comes from trusted runtime context. The search engine filters candidates to that tenant before relevance scoring; hiding foreign results after retrieval is too late because unauthorized data already crossed the boundary.

Distinguish “can” from “reliably does”

Stochastic systems need repeated trials. If a case passes once in five runs, the agent demonstrated occasional capability, not dependable behavior.

• pass@1 estimates success for one sampled trial.
• pass@5 asks whether at least one of five trials succeeds.
• pass^5 asks whether all five trials succeed.

The distinction is visible in a simple pattern: P F F F F has observed pass@1 of 0.2, pass@5 of 1, and pass^5 of 0. Report the pattern by task and risk alongside latency, tokens, cost, and tool errors. Five trials expose variance; they do not justify precise production-rate claims.

The τ-bench authors introduced repeated interaction evaluation for tool-using agents and highlighted reliability across runs (paper, repository). The general lesson is durable: one lucky trajectory is not a release argument.

Diagnose the earliest causal failure

Final symptoms misdirect repairs. A stale retrieved policy can cause a wrong proposal, rejected effect, and misleading final message. Fixing the message treats the last symptom, not the first cause.

Assign every failed trial one primary earliest cause:

1. intent misunderstanding;
2. planning;
3. retrieval;
4. tool selection;
5. tool arguments;
6. execution;
7. state interpretation;
8. recovery;
9. authorization or safety;
10. final communication;
11. evaluation defect.

The eleventh category is essential. Reference solutions, negative controls, grader calibration, and trace review can show that the agent succeeded while the task or evaluator was wrong. Evaluation systems need evaluation too.

OpenTelemetry spans provide a portable causal skeleton for model, retrieval, tool, approval, and state boundaries. Its semantic conventions and GenAI attribute registry evolve, so pin versions and keep recorded attributes bounded. Raw prompts, secrets, policy text, and personal data do not belong in general telemetry.

Calibrate subjective judges against people

LLM judges are appropriate for bounded subjective questions such as whether final communication is clear and appropriately qualified. They should not decide whether an unauthorized mutation occurred when executable state evidence exists.

Calibrate a judge by labeling trials blind to judge output, then publish the confusion matrix, false-pass and false-fail rates, disagreements, rubric, model, prompt version, and sampling settings. Research on G-Eval (paper) and MT-Bench judges (paper) shows both the utility and biases of model-based grading. Calibration is not optional ceremony; it tells you what the instrument can safely measure.

Close the loop through deployment

Offline evidence becomes operational only when it governs rollout. Compare candidate versions under identical suite, grader, policy, tool, model, and environment digests. Reject comparisons with missing repeated trials, changed thresholds, broken decisive instruments, or mismatched suites.

Then progress through bounded exposure:

flowchart LR
  suite["Frozen offline suite"] --> shadow["Shadow: observe, no effects"]
  shadow --> canary["Canary: bounded low-risk traffic"]
  canary --> rollout["Wider authorized effects"]
  rollout --> failure["Confirmed failure"]
  failure --> regression["New frozen regression case"]
  regression --> suite

Predeclare abort rules. Any critical safety failure, missing decisive evidence, high-risk outcome regression, reliability-floor breach, or cost/latency budget breach stops promotion. Roll back the exact candidate artifact and preserve evidence.

The 100-case Enterprise Access Agent reference uses 30 normal, 25 boundary, 20 failure, and 25 adversarial cases. At immutable tag v1.0.2, its deterministic offline suite report records pass rate 1.0 and zero critical failures; its five-trial report records suite digest 95bab483fd900af9f5b91c7e52f7c8b7227dbf839bb49755efb7877fcf20da93 and raw-results digest ada0f448a671389e4e0593c4890140abc09e4f0d2b56546f220b96c81bd658aa. Offline cost and tokens are zero, while zero latency is a placeholder—not measured provider performance. These are bounded reference results, not universal safety or production-readiness claims.

What to build first

Start smaller than a dashboard or framework migration:

1. Write the system contract and five tool authorization rules.
2. Write 20 cases before implementing the agent.
3. Build one deterministic workflow and record where autonomy is necessary.
4. Add a bounded decide → act → observe loop without moving authorization into prompts.
5. Capture final state and causal traces.
6. Add independent outcome, safety, trajectory, retrieval, and calibrated communication graders.
7. Repeat trials, predeclare release gates, and turn confirmed failures into regressions.

Use portable components such as Inspect AI, OpenTelemetry, and optional Phoenix inspection. OpenAI’s June 3, 2026 AgentKit update says hosted Agent Builder and Evals stop being available after November 30, 2026 (official notice). Learn durable evaluation concepts; do not make your evidence loop depend on a disappearing hosted surface.

Agentic systems do not become trustworthy when a benchmark number rises. They become governable when evidence separates what happened, how it happened, how often it happens, what it costs, why it failed, and whether that failure can recur after the next release.