Separate Outcomes, Trajectories, and Judges

Week 5 produced R5-suite-v1, 75 frozen cases, and reproducible harness logs. One scorer still cannot tell whether access changed correctly, safety held, the tool path was legal, and the explanation was clear. This week you separate those claims: executable outcome and safety graders inspect authoritative state, a rule grader inspects bounded trajectory facts, and an LLM judge evaluates only subjective communication.

A grader maps run evidence to a bounded result. Independence matters: a fluent explanation cannot repair a foreign-tenant effect, and an unconventional but legal tool path should not fail because it differs from one reference trace.

The runtime at agent.py emits evidence but never receives reference answers or grader output. This separation prevents evaluation-aware behavior.

The outcome grader checks the authoritative employee/resource state after execution. Safety graders assert zero cross-tenant disclosure/effect and zero unauthorized mutation. These are deterministic release gates, not percentages averaged with prose quality.

evals/task.py runs graders independently and preserves all results:

        scores = {
            "outcome": asdict(outcome_score),
            "safety": asdict(safety_score),
            "trajectory": asdict(score_trajectory(case, result)),
            "rag": asdict(score_rag(case, result)),
            "judge": asdict(score_communication(case, result)),
        }

Never produce one artificial overall quality score. Release treatment keeps hard safety, business outcome, quality, reliability, and operations separate.

Trajectory grading should protect necessary order and arguments without demanding one ideal path. Require tenant-safe lookup, approval before mutation, proposal/approval digest equality, a bounded retry count, and final verification. Allow extra read-only inspection when budgets permit.

The offline recorder at telemetry.py supplies allowlisted spans, but the tag does not yet wire full causal order through every tool. That wiring is a design target. Mutation tests should remove or reorder each decisive event; the relevant grader must catch every mutation.

An LLM judge is another fallible measurement system. Hand-label 30 trials without seeing judge output, then report a confusion matrix, false-pass rate, false-fail rate, disagreements, rubric, model, prompt version, and sampling settings. The judge assesses whether final communication is accurate, concise, and appropriately qualified; it cannot override deterministic failure.

The tagged calibration report explicitly says v1 uses deterministic communication-rubric-v1, not an LLM judge. Thirty blinded human labels, confusion matrix, false-pass rate, and false-fail rate remain a design target before any provider-backed judge can influence subjective quality. G-Eval introduced criteria-driven chain-of-thought evaluation, while the MT-Bench judge paper documents judge position and style biases; calibration against your task distribution is mandatory.

Run a correct restore and a missing-approval denial with Globex as unaffected positive control. Then feed the judge a polished message that falsely claims success. Outcome must fail regardless of judge preference. Recovery fixes the communication or its evidence—not thresholds after seeing results. Cleanup removes temporary judge responses while retaining labels and digests.

The scored checkpoint supplies one assertion; the unknown is which grader is decisive. Feedback contrasts trajectory preference with business/safety outcome. Decline any average that lets prose quality dilute a critical failure. Spend 1.5 hours reading OpenAI evaluation best practices and judge papers, 4–5 hours building/mutation-testing graders and labeling 30 runs, and 1 hour writing calibration limits.

The handoff is R6-graders-v1 plus calibration report. Week 7 consumes independent retrieval and generation evidence to diagnose RAG failures without changing both systems at once.