Generate and Revise Through a Visible Agent Run

Enterprise problem: A customer has approved what to build but cannot safely give an opaque agent broad repository, network, secret, or deployment access; without visible actions and finite authority, one wrong instruction can damage other projects or hide why the result changed.

Whole-course context: The incoming artifact is Day 01’s approved App Blueprint bp-1, including its digest, acceptance cases, archetype probes, budgets, and boundary map; this is the generate-and-revise milestone before interactive preview and managed data.

Today’s slice: The provider-neutral control plane starts one project-scoped agent run through an explicit execution-backend adapter, shows its plan, typed tool calls, patches, checks, budget, approvals, and stop reason, then accepts a customer revision without granting production authority.

End-of-day evidence: The customer receives source revision rev-2, a sandbox manifest, patch history, gate artifacts, and an immutable run trace tied to bp-1.

Still unsolved: Preview interaction, persistent app data, enterprise connectors, human identity, sharing, production publication, and fleet operation remain deliberately deferred.

The customer outcome is run coding agents as bounded sandbox jobs. This day makes the mechanism observable before returning to policy, failure handling, and evidence; it does not repeat a requirements catalogue.

Orchestrator owns leases and budgets; sandbox manager owns execution; tool broker owns effects; evaluator owns gates. Compute: ephemeral containers with CPU, memory, and network limits. Storage: PostgreSQL runs/revisions, object storage patches/logs, Redis leases/cancellation.

Use a finite plan–patch–check–observe state machine. Create a sandbox from an approved base digest, pass typed tools only, record every patch and gate, and terminate on budget, cancellation, or denial.

A provider SDK can make one sandbox demo look like the whole product, so a team may couple policy to vendor calls and overlook the customer outcome, lifecycle, trust boundaries, or recovery path. Follow the same Northstar Workboard system from Day 01 while adding one responsibility at a time; Daytona and Cloudflare remain adapters beneath outcome-based contracts, never interchangeable products.

Northstar has approved Workboard blueprint bp-1, but no running app exists. The full journey remains create → generate → preview/interact → managed data → enterprise connectors/private connectivity → identity → share/revoke → publish → change/redeploy/rollback → operate/support → export/retire/delete; today implements only generate-and-revise and hands an evidence-bound source revision to preview.

It preserves the complete create-to-retire customer journey while marking today’s exact input and output.

If a generation action cannot be tied to the approved blueprint, current lifecycle stage, later owner, and eventual cleanup obligation, stop before allocating compute.

Generation receives an immutable blueprint digest and returns source revision, run trace, gate artifacts, and cleanup proof. It neither creates a generated-app database nor converts preview, publication, operation, or retirement into implied permissions.

Northstar needs a dependable answer to “what turned bp-1 into rev-2?” before learning how sandboxes and brokers work. The smallest complete model is an approved contract, a bounded and observable generation loop, and a source revision whose eligibility is backed by evidence.

It shows the minimum complete transformation and both terminal outcomes before implementation detail.

A generated revision is eligible only when the exact input digest, bounded run, current-revision checks, stop reason, and immutable evidence form one causal chain.

“The model returned files” fails this rule because it says nothing about the approved scope, changed revision, denied actions, or why execution stopped.

The three-box model still hides where authority can cross planes, so Northstar project membership could be mistaken for Workboard access or a sandbox lease for enterprise-data permission. Expand it into provider/control-plane, hosted-runtime, generated-application, and connector/source boundaries whose grants are independent and revocable.

It assigns each kind of authority and data to its owning plane and makes every cross-plane binding explicit, least-privilege, audited, and independently revocable.

Never infer generated-app, connector, source-system, preview, or deployment authority from a provider actor, run token, or sandbox identity; verify and revoke each mapping separately.

Tenant isolation is an allocation rule as well as a provider feature: never multiplex mutually distrusting organizations or runs inside one sandbox merely because processes can be named separately. Provider isolation also does not supply product authentication, authorization, rate limits, approvals, evidence retention, or cross-plane mappings automatically.

Once the boundaries are explicit, the next failure mode is duplicated policy or a provider adapter that quietly becomes the orchestrator, secret broker, and evidence store. Expand the generation box into single-responsibility services, authoritative state, one SandboxPort, distinct provider adapters, and a DRY declarative capability profile.

It gives each service one reason to change, identifies authoritative stores, and confines provider-specific behavior to adapters under one measurable port.

Put shared invariants in versioned declarative contracts, keep orchestration and authorization outside adapters, and deny allocation when a provider capability is missing or unproven.

export type ExecutionBackend = SandboxPort;

export interface SandboxPort {
  readonly backendId: 'daytona' | 'cloudflare-sandbox';
  inspectCapabilities(): Promise<CapabilityManifest>;
  allocate(spec: SandboxSpec): Promise<SandboxLease>;
  writeFiles(lease: SandboxLease, batch: FileBatch): Promise<FileReceipt>;
  executeNamed(
    lease: SandboxLease,
    command: NamedCommand,
    signal: AbortSignal,
  ): Promise<ExecutionObservation>;
  exposePreview(lease: SandboxLease, policy: PreviewPolicy): Promise<PreviewGrant>;
  revokePreview(grant: PreviewGrant): Promise<RevocationReceipt>;
  cancel(lease: SandboxLease, reason: StopReason): Promise<CancellationReceipt>;
  destroy(lease: SandboxLease): Promise<DestructionReceipt>;
}

export interface CapabilityManifest {
  manifestVersion: 'v1';
  backendId: SandboxPort['backendId'];
  adapterVersion: string;
  isolation: 'dedicated-sandbox-per-run';
  egress: 'deny-all' | 'explicit-allowlist' | 'trusted-proxy';
  secretDelivery: 'opaque-provider-placeholder' | 'short-lived-proxy-token';
  preview: 'disabled' | 'authenticated' | 'possession-url-plus-app-auth';
  cancellation: 'process-stop-then-destroy';
  providerAudit: 'available' | 'not-relied-upon';
  evidenceProbeIds: readonly string[];
}

SandboxPort owns translation and normalization only. The run orchestrator owns the state machine; policy decides whether a call is permitted; the secret broker owns credentials; the evidence writer owns immutable records; the source service owns revisions. This is SRP. The same versioned execution profile, named-command catalog, policy bundle, and evidence schema are referenced by both adapters, which applies DRY without erasing real provider differences.

The capability manifest is measured at adapter startup and recorded with each lease. A missing or degraded required capability fails allocation closed; it is never filled with a hopeful default.

apiVersion: coding-agent.enterprise/v1
kind: ExecutionCapabilityProfile
metadata:
  name: secure-generation-v1
spec:
  required:
    isolation: dedicated-sandbox-per-run
    egress: deny-all
    secretDelivery: brokered-no-plaintext
    preview: disabled-until-explicit-grant
    cancellation: process-stop-then-destroy
    evidence: immutable-positive-and-negative-probes
  policyRefs:
    commands: policy://named-commands/v3
    network: policy://generation-egress/v5
    secrets: policy://secret-broker/v4
    retention: policy://run-evidence-retention/v2

Provider adapter translations and IaC overlays

Whole-system location: The complete lifecycle still consumes the same blueprint, source, policy, and evidence identifiers, while today’s lowest level maps the portable contract to actual Daytona or Cloudflare controls and proves both a successful run and a denied run.

The mapping is deliberately asymmetric. Daytona sandboxes expose lifecycle, filesystem, process, preview, secret, and audit facilities through Daytona APIs and SDKs; Cloudflare Sandbox SDK is controlled from a Worker and exposes command, file, process, port, and lifecycle operations. The adapter normalizes outcomes but never promises a capability that the chosen backend and account configuration cannot demonstrate.

A provider can accept an allocation call while failing to enforce the intended outcome, and a cancelled Workboard run can leave processes, exposure, or partial source behind. The complete system therefore exercises the customer-visible path through admission, generation, denial, cancellation, cleanup, recovery, and immutable evidence.

It turns the portable contract into observed positive, denied, failure, cancellation, cleanup, and recovery paths that all end in immutable evidence.

Admit a provider for this profile only when both the allowed Workboard path and every required denial/recovery probe reach the expected terminal state with a provider reference and platform-owned evidence.

An opaque “building” spinner hides scope drift, failed checks, and unsafe actions, so a customer cannot tell whether the agent followed the approved contract or merely produced plausible files. A visible agent run is a control-plane object whose phases, proposed actions, observations, limits, approvals, and outputs can be inspected while it runs and after it stops.

The run starts only from an approved blueprint digest and creates a fresh source revision in the project workspace. Its user-facing phases are queued, planning, editing, checking, needs-approval, completed, failed, or cancelled; those transitions are included in the customer-flow diagram above. The interface shows the current goal, changed-file list, remaining budget, last structured observation, and whether a person or policy must act. It never displays secrets or raw hidden reasoning.

“Completed” means a preview-ready source revision was produced under this run policy. It does not mean the app is validated, deployed, or safe for live users. A customer can cancel a run, but cancellation is cooperative: the orchestrator stops new calls, terminates sandbox processes, exports the trace, and marks any partially produced revision ineligible for preview until checks finish.

The primary lab generates the Workboard create-and-complete tracer bullet from bp-1, reviews a failed idempotency test, and requests one revision. A tracer bullet is a thin implementation through real layers, not the platform’s product limit.

A generic shell or “do anything” tool combines unrelated powers, so the runtime cannot permit a safe read while reliably denying secret access or deployment. A typed tool has a validated input and output schema, a side-effect class, a timeout, and a policy decision enforced outside the model.

tools:
  - name: read_project_file
    input: { projectId: project_id, revisionId: revision_id, path: relative_path }
    output: { content: string, sha256: digest }
    sideEffect: none
  - name: apply_project_patch
    input: { revisionId: revision_id, expectedBaseSha: digest, patch: unified_diff }
    output: { changedFiles: string_list, revisionSha: digest }
    sideEffect: project_workspace_write
  - name: run_named_gate
    input: { revisionId: revision_id, gateId: approved_gate, idempotencyKey: string }
    output: { exitCode: integer, artifactId: immutable_id, durationMs: integer }
    sideEffect: sandbox_process
  - name: request_scoped_approval
    input: { proposalId: immutable_id, capability: string, reason: string }
    output: { decision: approved_or_rejected, receiptId: immutable_id }
    sideEffect: none

The agent selects gateId: unit.todo-idempotency; it cannot submit arbitrary shell text. The trusted orchestrator maps that identifier to a pinned command, image, and resource profile. Paths are normalized, resolved under this project’s copy-on-write workspace, and rechecked after symbolic-link resolution. Outputs are size-limited structured data with an error class and artifact address.

Tool output is untrusted input. Repository instructions, test failures, dependency metadata, and fetched documentation can contain prompt injection, text intended to trick the model into seeking new authority. Such text can inform a patch but cannot grant capabilities, alter the blueprint, approve a change, or widen network policy.

Generated code and dependencies can be malicious or simply broken, so host execution with ambient files, credentials, and networks can turn a project mistake into an enterprise incident. A sandbox is a disposable execution environment that confines the run’s filesystem, processes, resources, and network.

Create the sandbox from a pinned image and run as a non-root principal. Mount only the project revision plus synthetic fixtures; make the operating-system filesystem read-only except for small workspace and temporary volumes. Enforce central-processing-unit, memory, disk, process-count, output-size, and wall-clock limits. Destroy the environment after exporting approved artifacts and trace records.

Network is denied by default. A controlled proxy may allow an exact package digest or approved documentation origin, then re-resolve the destination, reject redirects to loopback, link-local, metadata, or private ranges, cap bytes and time, and log the decision. A hostname allowlist alone is insufficient because name resolution and redirects can change the destination. The sandbox receives no production data, cloud credential, source-control write token, connector grant, or deployment authority.

Isolation limits blast radius but does not prove hostile code harmless. Patched hosts, process and syscall controls, dependency verification, output scanning, sandbox escape monitoring, and one-run credentials provide defense in depth.

One long generation pass makes it hard to identify the change that caused a failure, so recovery becomes guesswork and a customer sees only the final story. The orchestrator instead alternates a small proposal with a policy decision and a structured observation.

1. Pin projectId, blueprintDigest, base revision, allowed paths, named gates, and budgets.
2. Ask the agent for a short plan whose steps reference blueprint acceptance-case IDs.
3. Validate the next typed call; deny any missing, malformed, or ungranted capability.
4. Apply one coherent patch with an expected base digest to prevent lost updates.
5. Run the cheapest relevant named gate and attach its output to the new patch digest.
6. On failure, require a root-cause hypothesis and preserve the failed artifact before another patch.
7. Stop when required generation gates pass, a hard limit is reached, policy denies the action, the customer cancels, or approval expires.

The customer-flow diagram above preserves this plan–policy–patch–check–observe loop and both of its terminal branches. The model never sets acceptanceMet. Trusted policy derives the state from current-revision gate records. It must reject stale results from an earlier patch, changes to protected tests, deletion of an assertion, widened tenant predicates, ignore directives, or replacement of a deterministic oracle with model judgement unless a new blueprint version explicitly authorizes that scope.

An agent that can retry forever or quietly widen its file and dependency scope creates unpredictable cost and hides failed reasoning. Runtime-enforced budgets make failure finite and reviewable.

runBudget:
  maxModelSteps: 24
  maxPatchAttemptsPerGate: 4
  maxElapsedMinutes: 30
  maxChangedFiles: 12
  maxAddedDependencies: 1
  maxParallelProcesses: 2
  maxToolCostUsd: 4.00
  stopOnRepeatedFailureFingerprint: 2

A failure fingerprint classifies the gate, error type, and relevant location. Repeating it after two claimed fixes suggests the hypothesis is wrong; the run stops or requests a different diagnostic capability instead of making cosmetic edits. Only transient, side-effect-safe operations retry automatically. Writes carry an idempotency key so a network retry returns the original effect rather than applying it twice.

Local reads, patches, synthetic fixtures, and named checks may proceed under the approved run policy. Adding an unapproved dependency, accessing a new network origin, changing authentication configuration, weakening a gate, deleting a migration, publishing, merging protected code, deploying, changing production configuration, rotating a real secret, or sending an external message requires a separately classified capability and often remains prohibited.

Approval is proposal-specific, time-bound, single-use, and performed by a trusted broker. The model receives a receipt, never the privileged credential. A request includes the exact patch or artifact digest, target, expected effect, relevant evidence, and rollback or cancellation semantics. Ambiguous classification fails closed.

If the day exercises many unrelated builds, the evidence becomes broad but shallow and the customer cannot follow one causal run. This lab uses one Workboard workflow to cross blueprint, source, tests, and revision history; other archetypes remain bounded compatibility references.

1. Start run run-wb-017 against approved bp-1 and base revision rev-0; verify a draft blueprint is rejected.
2. Inspect the plan: domain transition, tenant-scoped repository operation, application programming interface (API), minimal controls, and tests. Reject any notification, analytics, identity, connector, or deployment work as out of scope.
3. Watch the agent patch the domain and run unit.todo-idempotency. Preserve the first failure showing a duplicate completion event.
4. Review the root-cause hypothesis, accept the ordinary source correction within existing policy, and rerun the same protected gate.
5. Watch tenant-integration and clean-build gates run against the current patch digest; verify an attempted read of another project is denied and recorded.
6. Ask, “Keep completion visible after a refresh.” The orchestrator creates a revision request tied to the same blueprint case; it does not silently rewrite the approved job.
7. Compare rev-1 with rev-2, inspect changed files and artifacts, then mark rev-2 preview-ready when all required generation gates pass.
8. Cancel a synthetic extra run and verify no new tool call begins, child processes stop, and its partial revision remains ineligible.

A single tracer bullet can accidentally bake Workboard assumptions into shared orchestration, so a reference check must challenge the platform contract beyond todos. Secondary archetype probes remain schema and policy tests; they do not distract from the day’s one executable customer workflow.

A run summary without exact actor, scope, precondition, artifact, time, and environment can be attached to the wrong project or revision, making review and incident reconstruction unreliable. Store one append-only row per material action and decision.

Agent policy can rest on incomplete threat assumptions, so reviewers need primary security and software-lifecycle sources that make controls challengeable. These official references support external policy enforcement, least privilege, provenance, and evidence without prescribing a particular commercial platform.

• Daytona documentation — official sandbox lifecycle, filesystem, process, preview, networking, and SDK reference entry point.
• Daytona secrets — official opaque-placeholder, HTTPS-header substitution, host allowlist, and response-scrubbing semantics.
• Daytona audit logs — official organization audit fields and access model; platform evidence still preserves its own immutable schema.
• Cloudflare Sandbox SDK — official Worker-controlled sandbox, command, file, process, and service-exposure entry point.
• Cloudflare Sandbox security model — official isolation boundary, within-sandbox sharing, preview-access, application-security, and cleanup responsibilities.
• Cloudflare proxy requests — official short-lived JWT and Worker-held credential-broker pattern.
• Cloudflare outbound traffic — official default internet disablement, allowed-host, and outbound-handler controls.
• NIST SP 800-218A — secure-development practices for generative AI and dual-use foundation models.
• NIST SP 800-218 — secure software development practices applicable to generated source and its surrounding lifecycle.
• NIST SP 800-207, Zero Trust Architecture — resource-centric, continuously evaluated access rather than trust from network location.
• SLSA specification — official supply-chain provenance levels and build-track requirements for later artifact evidence.

Detailed traces are useful only if the main safety decisions remain obvious to the customer and operator. Carry these constraints into the preview milestone.

• Generation is a visible, cancellable, revisioned control-plane workflow.
• Typed tools and external policy, not model instructions, form the capability boundary.
• Each run gets an ephemeral project sandbox, synthetic data, deny-by-default network, and no production authority.
• Finite budgets, protected oracles, and current-revision evidence determine when the loop stops.
• Workboard is one tracer bullet; reference probes challenge shared orchestration assumptions without claiming universal support.

A run that merely ends with files can conceal unsafe calls, stale checks, or lost failures, so completion requires inspectable causal evidence. Every checked item should resolve to a visible run event or immutable artifact.

• [ ] The run is pinned to organization, project, approved blueprint digest, base revision, and policy digest.
• [ ] The customer can see phase, plan, changed files, named gates, budget, approvals, and stop reason.
• [ ] Tool inputs and outputs are typed, bounded, and authorized outside the model.
• [ ] The sandbox has no ambient host, project, secret, source-write, connector, or deployment access.
• [ ] Network access is deny-by-default and destination-checked at connection time.
• [ ] Failures, denials, patches, and current-revision results remain in the trace.
• [ ] Retry, cost, time, file, dependency, process, and side-effect budgets are enforced.
• [ ] Approval is exact, single-use, expiring, and broker-executed.
• [ ] rev-2 is preview-ready evidence, not a production-release claim.

Continue Northstar's supplier-onboarding path in run-use-cases.ts. The orchestrator will accept only the digest projected from Day 01, then gives the run finite step and cost budgets.

public async start(context: RequestContext, command: StartRunCommand): Promise<Run> {
  if (
    !(await this.blueprints.isApproved(
      context.tenantId,
      command.blueprintId,
      command.blueprintDigest,
    ))
  ) {
    throw new BlueprintNotApprovedError('Approved blueprint digest required');
  }
  const runId = `run_${this.ids.next()}` as RunId;
  const run = new Run({
    tenantId: context.tenantId,
    runId,
    blueprintId: command.blueprintId,
    blueprintDigest: command.blueprintDigest,
    maxSteps: command.maxSteps,
    maxCostUsd: command.maxCostUsd,
  });
  await this.runs.save(run);
  return run;
}

The command declares a bounded run, not ambient permission to build anything. The application service interprets it, consults the approved-blueprint projection, constructs a domain state machine, and stores the run. Node consumes CPU and memory locally; a deployed worker would consume ECS task capacity plus database network and disk. A run snapshot with the pinned digest and enforced budget is evidence; an exception before save is denial evidence.

SRP assigns run lifecycle to the orchestrator while approval remains in the control plane. DRY shares typed commands and identifiers. IoC/DI inject the projection, repository, unit of work, clock, and ID generator. MVC leaves request translation to the controller. PubSub supplies the approval projection and later emits RunCompleted.v1; IaC creates the queues, tasks, and stores that host those ports.

pnpm vitest run services/run-orchestrator/src/run-orchestrator.test.ts
pnpm smoke:product

Try an unapproved digest, consume beyond maxSteps or maxCostUsd, cancel a run, and send the same run ID through a second tenant. The claim survives only if every attempt stops before a new effect, cancellation prevents further work, and the other tenant receives no existence signal. The repository proves orchestration state and budgets; sandbox process isolation remains a deliberate adapter boundary, not a result implied by these tests.