Close the Evaluation and Deployment Loop
Week 10 of 10 · Compare v1/v2, choose promote/hold/rollback, and convert failures into regressions • Public source target: https://github.com/ZGTR/enterprise-access-agent-evals/tree/v1.0.2
System map · Day 10
Whole-system design
Five stable layers. Today's work is expanded and linked; the rest stays in context.
Entry and authorization control
Covered — Request entry · Tenant and policy authority · Approval gate
Compute and execution
Covered — Policy retrieval compute · Typed tool broker
Agent runtime compute
Design target · not proved
Provides pinned v1 and v2 candidates under identical suite inputs.
Enterprise resource boundary
Covered — Simulated enterprise system
Storage and state
Covered — Run and effect state
Evaluation evidence store
Source-backed today
Admits confirmed production-like failures as new frozen regression cases.
Evidence and release control
Evaluation harness compute
Source-backed today
Rejects mismatched suites, changed thresholds, and missing five-trial evidence.
Trace and diagnosis plane
Design target · not proved
Connects each regression to decisive causal spans and component ownership.
Release gate
Design target · not proved
Produces promote, hold, or rollback evidence; live canary operation remains unproved.
Traversed today
Compare versions only on frozen ground
Week 9 left R9-security-gate, 100 frozen cases, and zero-tolerance attack gates. The final problem is operational: a result matters only if it changes release behavior, and production-like failures matter only if they improve the next suite. This week you compare v1 and v2 under identical dataset, grader, policy, tool, environment, and five-trial digests, then choose PROMOTE, HOLD, or ROLLBACK.
A closed loop is observation → diagnosis → decision → action → measurement → regression. It does not mean automatically training on every bad output. Human review confirms the earliest causal failure and expected behavior before a case becomes release authority.
Tagged agent.py supplies the v1 candidate. A real v2 result is not included; create one candidate change for this exercise, and abort comparison if suite, policy, model, tool, grader, or environment digests differ.
Reject incomparable or incomplete evidence
evals/compare.py rejects suite mismatch, requires five trials, and returns ROLLBACK, HOLD, or PROMOTE from predeclared evidence. Broken instruments and threshold immutability are additional exercise gates.
evals/dataset.py keeps safety, outcome, trajectory, RAG, reliability, cost, and diagnosis tables separate. Never compute one artificial overall quality score.
| Evidence | v1 | v2 | Release interpretation |
|---|---|---|---|
| Critical safety failures | 0 | 0 | required gate holds |
| Business outcome by risk | report slice | report slice | candidate must not regress high risk |
| pass^5 by risk | report slice | report slice | consistency, not occasional success |
| latency and cost | distribution | distribution | budget gate, not quality proxy |
| Earliest causes | counts + spans | counts + spans | directs repair ownership |
The immutable v1.0.2 suite report records 100 cases, pass rate 1.0, zero critical failures, agent/grader version 1.0.0, and suite digest 95bab483fd900af9f5b91c7e52f7c8b7227dbf839bb49755efb7877fcf20da93. This is bounded deterministic offline evidence, not a v1-versus-v2 result and not production proof.
Diagnose before admitting a regression
telemetry.py provides a recorder seam, but complete candidate-to-cause linking remains a design target. Confirm whether failure is intent, planning, retrieval, tool selection, tool arguments, execution, state interpretation, recovery, authorization/safety, final communication, or evaluation defect.
Only a confirmed reproducible failure becomes a new case. Add initial state, expected/prohibited effects, decisive evidence, reference solution, and negative control; freeze a new suite digest; rerun impacted tests and full suite. This is how production-like evidence improves future releases without silently moving thresholds.
Design shadow and canary gates honestly
Tagged release/policy.yaml names shadow, canary, and progressive rollout but does not implement them. Shadow execution observes real-shaped requests without applying effects. A canary exposes a small bounded slice to the candidate with predeclared abort rules. Neither is production-proved by a local course run.
Recommended order: offline frozen suite → shadow with no mutation → canary for low-risk reads/proposals → narrowly authorized effects → broader rollout. Abort on any critical safety failure, missing decisive evidence, high-risk outcome regression, reliability floor breach, or cost/latency budget breach. Roll back the exact candidate artifact; preserve evidence.
Run a comparable v1/v2 result where v2 improves task quality but introduces one tenant leak; decision is REJECT regardless of averages. Use an unaffected positive control for an approved Acme request. Recovery fixes the cause, adds regression, reruns all five trials, then produces a new decision. Cleanup disables disposable rollout flags and restores baseline state.
The scored checkpoint supplies frozen tables; the unknown is promote, hold, or rollback. Feedback contrasts metric movement with causal and comparability confidence. Decline changed thresholds after observing the candidate. Spend 1.5 hours on Anthropic eval guidance, SWE-bench, and Inspect, 4–5 hours on comparison/regression/rollout policy, and 1 hour producing the release memo, article outline, and 20-minute talk.
The accepted capstone is R10-release-decision, tagged public reference source, 100-case suite, five-trial evidence, article, and talk package. You can now build and teach an evaluation system that governs agent change instead of decorating a demo with one score.