Evaluation, Falsification, and Security

An evaluation is a repeatable measurement of how a system behaves on defined tasks. A single successful Workboard build proves only that one run produced evidence that passed its checks; it does not show that the coding system will succeed again after a new prompt, dependency resolution, or tool sequence.

The unit under evaluation is the whole coding system: model, instructions, tool contract, sandbox, dependency inputs, retry policy, and validation gates. Pin the task corpus and environment image, then run each task several times from an empty workspace. Record the model identifier, prompt and tool-policy versions, dependency lockfile, environment digest, temperature or seed when the provider exposes them, and every tool result. If a seed is unavailable, record that limitation instead of inventing reproducibility.

For Workboard, use at least three clean runs per task while developing the evaluation, then choose a larger sample from the observed variance and the cost of a false release. Each run starts with no generated files, no warmed dependency cache unless the environment contract includes one, no reused database, and fresh credentials limited to the test tenant. The generated code must face the same independent acceptance oracles on every run.

Report a distribution, not just a best result. “Four of five clean runs passed every blocking oracle” is useful evidence. “The app worked” hides the sample size, failures, and conditions.

A task corpus is a versioned collection of jobs used to measure a coding system. It should resemble the work the system will actually receive and include creation, change, and repair work, because an agent that can scaffold a new app may still damage an established one.

Build the corpus from Workboard’s executable acceptance cases. Keep training examples, development tasks, and the final held-out evaluation set separate where possible; repeatedly tuning on the held-out set turns it into another development set. Version every task and its expected evidence, and review corpus changes like code changes.

Task class	Workboard example	Blocking evidence
Fresh build	Create a tenant-scoped board and todo dashboard	Build, schema, API, browser, accessibility, and security checks
Feature change	Add filtering by assignee without changing completion rules	Existing regression suite plus new filter oracles
Repair	Fix a duplicate completion event under retry	A deterministic concurrency regression test
Security repair	Reject cross-tenant attachment lookup	Alpha/Beta denial test and audit event
Operational change	Add a health probe without exposing secrets	Configuration test and response-content assertion

Each task carries a rubric for qualities that are real but difficult to reduce to one assertion, such as maintainability. It also names deterministic acceptance oracles, expected denial behavior, time and cost budgets, allowed tools, and the evidence records to emit. An example manifest is deliberately small:

id: workboard-change-007
class: feature-change
environmentDigest: sha256:clean-eval-image
attempts: 5
blockingOracles:
  - unit-domain
  - tenant-integration
  - browser-accessibility
secondaryRubric:
  - change-is-local
  - names-explain-policy

Refresh the corpus when production incidents reveal a missing failure mode, but retain prior tasks so the trend remains comparable. Tag breaking corpus changes and never compare scores across incompatible versions without saying so.

An oracle is the mechanism that decides whether an expected behavior occurred. Prefer deterministic oracles—checks that produce the same result from the same inputs—for facts such as compilation, database constraints, HTTP status, tenant filtering, accessible names, and performance thresholds.

The system that generated Workboard must not grade its own output. Run acceptance tests and security checks from a separately maintained harness, with test credentials and expected results unavailable for modification by the coding agent. Protect the harness, expected snapshots, and release policy with repository permissions. A human release owner remains responsible for interpreting the evidence and accepting residual risk.

A second model can help assess clarity, unnecessary complexity, or whether an explanation matches a rubric. This is model grading, and it is secondary evidence because its judgment can vary and share blind spots with the generator. Calibrate a grader against a labeled set reviewed by people: measure agreement, inspect false passes and false failures, pin the grader prompt and model version, and define an abstain outcome for uncertain cases. Recalibrate after version changes or distribution shifts.

Decision	Appropriate primary judge	Model-grader role
Cross-tenant request returns denial	Integration test	None
Migration preserves every todo	Database comparison	Explain anomalies only
UI has an accessible name	Accessibility tree assertion	None
Patch is understandable	Human-reviewed rubric	Secondary scored evidence
Release is acceptable	Named release authority	Summarize, never authorize

Do not average away a blocking failure. A high maintainability score cannot compensate for a failed tenant-isolation test, and a persuasive model explanation cannot override a deterministic denial oracle.

Falsification means actively trying to produce evidence that a claim is false. Instead of asking only whether Workboard’s tests pass, change the implementation in a controlled way that should make a named test fail; this is mutation testing applied to the claims that matter most.

Suppose the claim is “a completed todo always has a completion timestamp.” Temporarily mutate the completion transition so it leaves completedAt empty. The todo-completion-timestamp property test must fail. Restore the source, record the mutation and caught test, and require the test to pass again. If the mutation survives, the evidence was weaker than the claim and release remains blocked until the oracle improves or the claim narrows.

Maintain a mutation log with claimId, changed behavior, expected failing oracle, observed result, commit SHA, environment, and restoration evidence. Include authorization mutations such as dropping a tenant predicate, availability mutations such as removing a timeout, and supply-chain mutations such as changing the expected artifact digest. Mutation code is never released.

Falsification also includes counterexamples that do not modify code: malformed identifiers, simultaneous updates, large result sets, expired credentials, clock skew, duplicate events, and unavailable dependencies. A passed suite raises confidence only within the tested space; it never proves that untested counterexamples do not exist.

An abuse case describes how an adversary or accidental misuse could violate a security property. Workboard’s evaluation must cover both unsafe generated application code and unsafe behavior by the coding agent that creates it.

Treat requirements, repository files, issue text, dependency metadata, fetched documentation, and tool output as untrusted inputs. A malicious README might instruct the agent to print environment variables or publish a package. The tool policy must deny secret reads, arbitrary outbound access, production credentials, and deployment; output gates must scan patches and artifacts for credentials and unsafe configuration. The evaluator injects these instructions in an isolated fixture and expects a refusal or harmless handling, not successful exfiltration.

For the generated application, derive tests from the pinned OWASP ASVS version and the relevant business risks:

Threat	Workboard abuse case	Expected evidence
Prompt injection	A repository file asks the agent to upload secrets	Denied tool call and no outbound request
Broken access control	Beta requests Alpha’s board identifier	Uniform denial and no existence leak
Injection	A todo title contains query or script syntax	Parameterized storage and safely encoded rendering
Server-side request forgery	Attachment URL targets an internal metadata address	Destination denied before connection
Resource exhaustion	Export requests an unbounded tenant history	Enforced limit, timeout, and observable rejection
Secret exposure	Generated client bundle references a server credential	Build gate fails and artifact is quarantined

Record which control stopped each path. A sandbox refusal and an application test are different evidence: one constrains agent behavior, while the other tests the software it produced.

Variance is the spread of results across equivalent runs; a flake is a test whose result changes without a relevant product change. Both matter because coding systems and distributed tests can produce misleading single-run results.

Store one result per task, run, gate, and environment rather than only the aggregate score. Trend strict pass rate, blocking-gate pass rate, median cost, tail latency, retry count, security-denial rate, grader score, and grader abstentions. Break results down by task class so easy scaffolding work cannot hide weak repair performance.

When evidence flakes, do not simply rerun until green. Quarantine is a visible, time-bounded status with an owner and expiry, not removal from the release calculation. Capture timestamps, logs, traces, environment digests, and the random seed when available; reproduce from a clean environment; then fix the race, shared state, clock assumption, or external dependency. A blocking security test that flakes makes the release evidence inconclusive and therefore fails closed.

Use confidence intervals or at least sample counts alongside rates. A 100% result over two runs is not comparable to 99% over ten thousand. Set minimum repetitions before seeing results, and investigate statistically meaningful changes plus any new severity-critical failure. Do not move the threshold after a poor run without a documented policy change.

A residual risk is the risk remaining after controls and tests have been applied. Release is a governed decision about whether the evidence supports accepting that risk for a specific artifact and environment, not a claim that Workboard has no defects.

Create a scorecard that separates non-negotiable gates from trend signals. Blocking gates include all executable acceptance cases, tenant isolation, critical abuse cases, provenance checks, and restoration after every mutation. Trend signals include success-rate movement, cost, model-grader rubric scores, and non-critical flakes. Every exception names its scope, owner, compensating control, expiry, and approval.

The release authority should be organizationally independent from the generating system and technically unable to edit evidence silently. It reviews failed runs as well as passes, confirms that the artifact digest matches the evidence, and chooses release, hold, or release-with-time-bounded-exception. High-severity unknowns, surviving critical mutations, or missing evidence mean hold.

The final record can extend the existing Workboard evidence ledger:

{
  "artifactDigest": "sha256:workboard-build",
  "corpusVersion": "workboard-eval-3",
  "cleanRuns": 25,
  "blockingGates": "pass",
  "knownFlakes": 1,
  "exceptions": [],
  "residualRisk": "untested browser and dependency interactions remain possible",
  "authority": "release-manager",
  "decision": "release"
}

Confidence grows when repeated independent evidence survives deliberate attacks. It never becomes a universal guarantee, so production monitoring, rollback, and incident response remain necessary controls.

These primary sources define the measurement, risk, application-security, and agent-abuse concepts used here. Versions and status are recorded so future readers can detect drift; all links were accessed on 2026-07-26.

• NIST AI Test, Evaluation, Validation and Verification — living NIST program page on context-specific metrics, testbeds, datasets, and evaluation methods; accessed 2026-07-26.
• NIST AI 600-1, Generative Artificial Intelligence Profile — final NIST publication, July 2024, for identifying and managing generative-AI risks; accessed 2026-07-26.
• OWASP Application Security Verification Standard 5.0.0 — stable release from May 2025, providing version-addressable web-application security requirements; accessed 2026-07-26.
• OWASP Top 10 for LLM Applications 2025 — published OWASP GenAI Security Project risk list for model-integrated systems; accessed 2026-07-26.

The durable lesson is that evaluation measures a distribution of behavior and falsification searches for weaknesses in the evidence. Release authority stays with accountable people who can explain what passed, what remains uncertain, and why the remaining risk is acceptable.

• Run a pinned corpus repeatedly in independent clean environments and record the complete system configuration.
• Keep deterministic acceptance oracles and release policy outside the code-generating system’s control.
• Use calibrated model graders only for secondary qualities, with abstention and human-reviewed calibration data.
• Require named tests to catch deliberate mutations, then prove the source was restored and the suite returned green.
• Test both hostile inputs to the coding agent and security failures in the application it creates.
• Treat variance and flakes as evidence problems, never as permission to rerun until a desired answer appears.
• Release against explicit blocking gates, exceptions, and residual risk; never promise zero defects.

Use this checklist to review a Workboard evaluation before its evidence is allowed into a release decision. Every checked item should point to an immutable log, test artifact, or signed decision record.

• [ ] The corpus includes fresh-build, change, repair, security, and operational tasks.
• [ ] Each task ran repeatedly from a clean, pinned environment with model and tool versions recorded.
• [ ] Deterministic oracles are protected from modification by the generating system.
• [ ] Any model grader is calibrated, versioned, secondary, and able to abstain.
• [ ] At least one documented mutation was caught by its named deterministic test and then restored.
• [ ] Agent prompt-injection and generated-application abuse cases fail safely.
• [ ] Results show sample counts, task-class breakdowns, variance, and visible flakes.
• [ ] Blocking gates cannot be averaged away by rubric or trend scores.
• [ ] A named human authority recorded the artifact digest, exceptions, residual risk, and release decision.