Distinguish Capability from Reliability
Week 8 of 10 · Run every one of 85 cases five times and keep consistency, cost, and latency separate • Public source target: https://github.com/ZGTR/enterprise-access-agent-evals/tree/v1.0.2
System map · Day 08
Whole-system design
Five stable layers. Today's work is expanded and linked; the rest stays in context.
Entry and authorization control
Covered — Request entry · Tenant and policy authority · Approval gate
Compute and execution
Covered — Policy retrieval compute · Typed tool broker
Agent runtime compute
Source-backed today
Runs each frozen case five times under pinned model and tool versions.
Enterprise resource boundary
Covered — Simulated enterprise system
Storage and state
Covered — Run and effect state
Evaluation evidence store
Source-backed today
Stores trial-level outcomes and task and risk slices under immutable digests.
Evidence and release control
Evaluation harness compute
Source-backed today
Computes pass at one, pass at five, pass to the fifth, variance, latency, tokens, and cost.
Trace and diagnosis plane
Design target · not proved
Measures per-trial operational evidence without substituting cost for quality.
Release gate
Source-backed today
Defines reliability and budget thresholds; rollout execution remains unproved.
Traversed today
One success does not make a dependable agent
Week 7 produced 85 frozen cases and separated retrieval from generation. A stochastic agent can pass once and fail the next four identical trials, so a single run measures neither repeatability nor operational cost. This week you run every case five times under pinned versions and report capability and reliability as different questions.
pass@1 is the probability one sampled trial passes. pass@5 asks whether at least one of five passes—useful for “sometimes capable.” pass^5 asks whether all five pass—useful for “consistently reliable.” State assumptions and sample limits; five trials reveal instability but do not establish a precise production rate.
agent.py runs each frozen input under the same agent, model, tool, policy, and budget versions. Each trial starts from independently reset state.
Preserve trial-level results before summarizing
evals/dataset.py stores all 425 trial records, not only averages. Slice by task type and risk so easy normal cases cannot hide flaky approval or tenant-boundary cases.
telemetry.py defines allowlisted latency, token, and cost attributes. The deterministic v1.0.2 report records zero placeholders for them; provider-backed measurement remains a design target. These are operability dimensions, never substitutes for outcome or safety.
Calculate three different reliability views
evals/task.py groups outcomes without inventing one overall score:
| Pattern across five trials | pass@1 estimate | pass@5 | pass^5 | Interpretation |
|---|---|---|---|---|
| P P P P P | 1.0 | 1 | 1 | consistently passed sample |
| P F F F F | 0.2 | 1 | 0 | sometimes capable, unreliable |
| F F F F F | 0.0 | 0 | 0 | no observed capability |
Report variance, median and tail latency, token/cost distribution, and tool-error rate alongside these outcomes. The τ-bench paper motivates repeated interaction evaluation; your release rule still follows the access agent’s own risk slices.
The final tagged offline reference runs ahead of this week’s staged 85-case milestone: v1.0.2 reliability evidence covers 100 cases × five deterministic trials, reports pass rate 1.0, zero critical failures, zero offline cost/tokens, suite digest 95bab483fd900af9f5b91c7e52f7c8b7227dbf839bb49755efb7877fcf20da93, and raw-results digest ada0f448a671389e4e0593c4890140abc09e4f0d2b56546f220b96c81bd658aa. Its zero latency values are placeholders, not measured provider performance.
Set budgets before comparing versions
Tagged release/policy.yaml predeclares five trials, zero critical failures, digest equality, and a zero-cost offline budget. Provider latency and risk-specific reliability floors remain design targets. A safety failure remains an immediate reject even if all averages improve.
Run a stable normal case and a five-run missing-approval denial, with Globex as unaffected control. Then inject a flaky timeout on one trial. Recovery fixes or bounds the timeout, reruns all five under a new agent version, and preserves both result sets. Cleanup stops all child processes and resets trial stores while retaining immutable raw-result digest.
The scored checkpoint supplies a five-outcome pattern; the unknown is the statistic that answers “sometimes capable” or “consistently reliable.” Feedback contrasts one success in five with five successes in five. Decline claims of statistical significance without intervals or sample caveats. Spend 1.5 hours on τ-bench and GAIA, 4–5 hours on property tests plus 425 trials, and 1 hour on risk-sliced interpretation.
The handoff is R8-reliability-baseline and a release budget. Week 9 turns critical security properties into hard, non-averagable release gates.