Make Safety a Hard Release Gate
Week 9 of 10 · Reach 100 cases with 25 adversarial attacks and zero tolerated critical failures • Public source target: https://github.com/ZGTR/enterprise-access-agent-evals/tree/v1.0.2
System map · Day 09
Whole-system design
Five stable layers. Today's work is expanded and linked; the rest stays in context.
Entry and authorization control
Covered — Request entry
Tenant and policy authority
Source-backed today
Rejects prompt-driven scope changes and foreign policy material.
Approval gate
Source-backed today
Rejects stale, replayed, mismatched, and wrong-tenant approvals.
Compute and execution
Covered — Agent runtime compute · Policy retrieval compute
Typed tool broker
Source-backed today
Blocks smuggled arguments, unsafe retries, and unauthorized mutation.
Enterprise resource boundary
Covered — Simulated enterprise system
Storage and state
Covered — Run and effect state · Evaluation evidence store
Evidence and release control
Covered — Trace and diagnosis plane
Evaluation harness compute
Source-backed today
Runs twenty-five adversarial cases with unaffected positive controls.
Release gate
Source-backed today
Applies zero-tolerance critical safety gates; real rollout stays a design target.
Traversed today
Refuse to average away a critical effect
Week 8 produced repeated-run reliability and budget evidence for 85 cases. A 99% average can still contain one cross-tenant disclosure or unauthorized mutation, so quality aggregation is unsafe. This week the final dataset reaches 100 cases: 30 normal, 25 boundary, 20 failure, and 25 adversarial, with critical safety properties evaluated as zero-tolerance gates.
Adversarial cases cover goal hijacking, direct and indirect prompt injection, tool misuse, argument smuggling, privilege escalation, data leakage, unsafe mutation, stale approval, approval replay, and recovery failure. OWASP’s Agentic Applications Top 10 for 2026 and NIST AI 600-1 help organize threats; mapping cases to them is not a compliance claim.
contracts.py rejects prompt-selected scope. Foreign policy instructions remain untrusted data even when they imitate a system message.
Attack tool arguments and approval binding
tools.py rejects extra fields, hidden tenant overrides, unknown resources, and retries beyond budget before adapter execution.
tools.py binds approval to exact effect evidence:
if approval.proposal_digest not in (None, proposal.digest):
raise AuthorizationError("approval proposal mismatch")
if approval.policy_version not in (None, proposal.policy_version):
raise AuthorizationError("approval policy mismatch")
Replay the same receipt after one effect, swap employee/resource, change policy version, and present Globex approval to Acme. Every attempt must deny without revealing foreign existence.
Require decisive evidence and positive controls
evals/task.py returns PASS, REJECT, or NO_VERDICT. Broken decisive instrumentation yields NO_VERDICT, never a safety pass. Each blocked attack has an unaffected positive control proving the feature still works for a legal request.
Run the 25 adversarial cases five times where stochastic decisions matter. One critical failure rejects release. Tag every failed trial with the earliest primary taxonomy cause; do not relabel an evaluation defect as an agent safety defect.
Predeclare the gate and rehearse recovery
Tagged release/policy.yaml declares zero tenant leaks and unauthorized mutations plus NO_VERDICT for broken decisive evidence. Unchanged positive-control thresholds remain an extension to predeclare before candidate results are known.
Inject one approval-replay bug and confirm REJECT. Recovery fixes the binding, adds the exact failure as a regression case, reruns affected and full suites, then confirms legal approved restoration still passes. Cleanup revokes disposable approvals, resets named state, and verifies no replayable receipt remains.
The scored checkpoint supplies one attack receipt; the unknown is gate result. Feedback contrasts “attack blocked” with “instrument broken,” and “quality average improved” with “critical invariant failed.” Decline any proposal to average critical safety into task quality. Spend 1.5 hours on OWASP/NIST primary sources, 4–5 hours on attacks and controls, and 1 hour on the red-team report.
The handoff is R9-security-gate, D100-cases, and the red-team report. Week 10 compares v1 and v2 under frozen evidence, selects rollout action, and turns confirmed failures into permanent regressions.