Freeze a Portable Evaluation Harness
Week 5 of 10 · Expand to 75 cases and run dataset → solver → scorers from one command • Public source target: https://github.com/ZGTR/enterprise-access-agent-evals/tree/v1.0.2
System map · Day 05
Whole-system design
Five stable layers. Today's work is expanded and linked; the rest stays in context.
Entry and authorization control
Covered — Request entry · Tenant and policy authority · Approval gate
Compute and execution
Covered — Policy retrieval compute · Typed tool broker
Agent runtime compute
Source-backed today
Acts as the solver target while the portable harness stays outside runtime.
Enterprise resource boundary
Covered — Simulated enterprise system
Storage and state
Run and effect state
Source-backed today
Resets disposable initial state for every independent case.
Evaluation evidence store
Source-backed today
Owns seventy-five versioned cases, references, negative controls, and suite digest.
Evidence and release control
Covered — Trace and diagnosis planeAhead — Release gate
Evaluation harness compute
Source-backed today
Runs frozen dataset, solver, and scorers from one command.
Traversed today
Turn manual checks into reproducible experiments
Week 4 left R4-trace-schema, 45 cases, and a causal failure ledger. Manual review found useful failures, but changing fixtures or thresholds between runs can make two agent versions incomparable. This week you freeze a portable evaluation harness: a dataset supplies cases, a solver runs the candidate, and multiple scorers evaluate independent claims.
Portable means the case schema and core scorers run without one hosted evaluation product. Use a framework-neutral CLI as authority and an Inspect AI adapter for orchestration. OpenAI’s hosted Agent Builder and Evals are scheduled to stop being available after November 30, 2026, according to its June 3, 2026 update; durable concepts must not depend on that hosted surface.
The candidate runtime at agent.py is only the solver target. It cannot read reference answers, expected effects, or grader thresholds.
Specify cases that can prove and disprove success
Every case pins tenant, actors, policy/source versions, initial state, request, available tools, approval state, expected outcome, prohibited effects, decisive evidence, class, risk, and reference solution. A suite digest covers ordered case content and schema version; editing one byte creates a new suite.
state.py resets named disposable state before each case. Isolation between trials prevents an earlier successful mutation from making a later case pass accidentally.
The frozen loader in evals/dataset.py rejects empty cases, missing decisive assertions, duplicate IDs, mutable thresholds, and references that fail their own case.
def load_cases(path: Path = CASES_PATH) -> tuple[EvalCase, ...]:
cases = tuple(
EvalCase.from_dict(json.loads(line)) for line in path.read_text().splitlines() if line
)
if not cases:
raise ValueError("dataset must contain at least one case")
if len({case.case_id for case in cases}) != len(cases):
raise ValueError("dataset case IDs must be unique")
return cases
Grow to 75 cases across normal, boundary, failure, and adversarial classes. Do not chase equal counts yet; preserve risk coverage and record denominators.
Run one command without creating one score
A harness coordinates evidence but must not collapse it. Business outcome, hard safety, task quality, trajectory, RAG, reliability, operations, and diagnosis remain separate outputs.
The tagged framework-neutral runner in evals/task.py preserves independent result lanes. An optional Inspect adapter should mirror this authority rather than replace it:
scores = {
"outcome": asdict(outcome_score),
"safety": asdict(safety_score),
"trajectory": asdict(score_trajectory(case, result)),
"rag": asdict(score_rag(case, result)),
"judge": asdict(score_communication(case, result)),
}
Run twice against deterministic adapters. Frozen inputs and deterministic solver should produce identical result digests. A model-backed solver may vary, but case, policy, tool, agent, grader, and environment versions must remain pinned so variance is interpretable.
Repair a vacuous case and preserve controls
Execute a normal restore and an unauthorized mutation denial with an unaffected Globex positive control. Then introduce a case whose expected final state already matches initial state. A do-nothing solver may pass, revealing a vacuous evaluation. Recovery changes initial state or adds decisive trajectory/effect evidence, proves the reference passes, and proves the no-op negative control fails.
Broken decisive instrumentation returns NO_VERDICT; it never becomes a safety pass. Cleanup resets each disposable state store and waits for every child/process exit while retaining immutable result digests.
The scored checkpoint gives one ambiguous case; the unknown is the minimal repair. Expected evidence is one-dimensional: reference passes and named negative control fails. Feedback shows that a 0% or 100% result can reveal a broken task or evaluator rather than agent quality. Decline proposals that add an agent feature while repairing the case.
Spend 1.5 hours on Anthropic’s agent eval guide and the Inspect tutorial, 4–5 hours on schema/harness/cases, and 1 hour documenting reproducibility. The handoff is R5-suite-v1, D75-cases, suite digest, and two harness logs. Week 6 adds independent graders and calibrates the only subjective judge.