Secure Delivery, Promotion, and Rollback

Enterprise problem: A customer can test a preview successfully yet publish different bytes, bindings, or database assumptions, turning “Publish” into an unreviewed rebuild with no safe recovery path.

Whole-course context: Days 03–04 produced a revision-bound stateful preview and governed connectors, while the companion authorization course defines separate identities and sharing/revocation evidence; today promotes those exact tested inputs rather than recreating them.

Today’s slice: The customer releases v1, attaches a stable custom domain, responds to production feedback with a backward-compatible v2 migration, canaries v2, rolls back on a threshold breach, fixes it, and redeploys.

End-of-day evidence: Digest-bound v1/v2 manifests, provenance, SBOM, migration records, domain/TLS checks, canary observations, rollback traces, and corrected-redeploy results all carry immutable artifact or run IDs.

Still unsolved: Cross-fleet SLO ownership, incidents, break-glass, backup restoration, residency, retention, export, deletion, and retirement remain deferred.

The customer outcome is a reliable, reviewable implementation of 05 secure delivery promotion and rollback. This day introduces the mechanism before policy detail and evidence review; it does not repeat the same customer stories in prose, tables, and diagrams.

Control-plane service owns desired state; runtime workers own execution; the policy/release boundary owns privileged effects. Compute: isolated service or sandbox tasks. Storage: PostgreSQL owns durable state, Redis is a bounded cache, object storage keeps artifacts/evidence, and queues are delivery buffers rather than authority.

Implement the topic as an owned state transition with a named controller, durable record, bounded worker action, and observable terminal evidence. Verify a denied or failed path before calling the mechanism complete.

A publish button hides five different truths: artifact bytes, runtime bindings, customer traffic, database schema, and rollback state. If one receipt conflates them, a green deployment can conceal a rebuilt artifact, a preview secret in production, a partially shifted route, an incompatible schema, or a destructive recovery. The diagrams below reveal one Workboard delivery system without changing its boundary.

Workboard begins as governed customer intent and ends only after export, retirement, and deletion proof. Day 5 owns the highlighted publish-to-recovery slice; agent execution, app authorization, source-system grants, and long-term operations keep their existing owners.

It fixes today’s location in the full create-to-retire lifecycle and defines success as a customer-observed production and recovery outcome, not a successful build job.

Never optimize a release step until you can name its upstream accepted evidence and its downstream lifecycle obligation.

The smallest complete model has exactly three responsibilities: accept immutable tested facts, promote them without reinterpretation, and observe both service and denial outcomes. Workboard v1 enters as an approved digest; the output is not “deployed” but a route, compatible data state, and evidence that customers receive those exact bytes.

Artifact, binding, traffic, schema, and rollback truth are separate even inside this minimal model: accepted facts identify bytes; promotion creates bindings, deployments, migrations, and routes; observations prove which combination served users and what recovery changed.

It introduces the smallest system that can turn an accepted Workboard artifact into a customer-verifiable production outcome and a bounded recovery decision.

Treat “publish succeeded” as false unless immutable input, controlled transformation, and observed terminal result all join through stable IDs.

The three-box model becomes unsafe when a sandbox token, deployment credential, app session, or connector grant crosses a boundary by implication. The provider control plane coordinates intent and evidence; an isolated workspace executes untrusted generation; the hosted runtime owns workload and route reconciliation; Workboard owns app users and tenant data; enterprise sources retain separate authority.

Each arrow is a typed, least-privilege contract with an independently revocable credential and audit event. Daytona sandboxes or the Cloudflare Sandbox SDK can implement isolated-workspace capabilities; neither becomes the authority for the customer organization, release policy, Workboard roles, database, connector grants, or evidence ledger. Cloudflare documents quick tunnels as ephemeral development exposure, so a stable production domain remains a separately governed runtime binding.

Boundaries alone do not prevent one privileged release service from rebuilding bytes, migrating data, and moving traffic. Single-responsibility principle (SRP) gives each service one owned decision; a versioned contract package keeps shared schemas DRY; declarative infrastructure as code (IaC) pins modules and parameters so observed state can be compared with reviewed intent.

The full design earns trust only when Workboard exercises the same system through success and refusal. Candidate 1 expands review_status, fails a predeclared journey threshold, and routes back to v1 without contracting the compatible schema; corrected candidate 2 receives new source, artifact, release, deployment, route, and test identities.

If Publish compiles mutable source again, the production result can differ from the preview the customer accepted. HelixWorks therefore builds once in an isolated builder, identifies the output by cryptographic digest, tests that digest, and promotes the same immutable artifact through environments.

Environment-specific configuration remains outside the artifact but is versioned in a release manifest. Promotion verifies the digest, signature, provenance predicate, software bill of materials (SBOM), acceptance evidence, policy, and approvers before creating a deployment. It never trusts a tag such as latest or a filename as identity.

A dashboard can show green checks from another revision or environment, so the admission decision needs one immutable join key. The release manifest binds source, build, artifact, tests, data changes, runtime bindings, policy, and customer approval.

{
  "releaseId": "rel-workboard-v1",
  "sourceRevision": "rev-3",
  "artifactDigest": "sha256:8a94...c12e",
  "buildRunId": "build-07-001",
  "provenanceId": "att-07-001",
  "sbomDigest": "sha256:26de...91aa",
  "acceptanceRunId": "accept-06-014",
  "migrationSet": ["m001", "m002"],
  "bindingSet": "bindings-prod-17",
  "policyVersion": "release-policy-9",
  "decision": "release"
}

The admission controller verifies that evidence was observed against the same digest or an explicitly declared pre-deployment condition. It rejects stale scans, unsigned or unexpected provenance, an unapproved builder, unresolved critical findings, missing migration compatibility proof, expired exceptions, and mutable dependencies. A human release, release-with-time-bounded-exception, or hold decision remains attributable and scoped.

An exact artifact can still behave incorrectly if production silently inherits preview databases, tokens, or networks. Runtime bindings are environment-owned references to secrets, databases, object storage, workload identity, connector grants, egress policy, telemetry, scaling, and domains; they are validated separately from artifact identity.

The production binding set names secret references and versions, never secret values. A deployment receives a new production workload identity and can request only production-scoped credentials. Day 4’s connector definition may be promoted by an administrator, but preview and production grants remain distinct; the companion authorization course keeps user delegation user- and source-scoped. The publish action does not copy refresh tokens, preview data, customer source data, or generated-app sessions.

Run preflight checks from the production network and identity: resolve required endpoints, authenticate through the broker, execute a harmless authorized source probe, deny unapproved destinations, verify database role and tenant predicates, and confirm telemetry intake. Network reachability is not authorization, and successful secret resolution is not permission to use every resource behind that secret.

Rolling back code while a destructive schema change remains can make the prior artifact fail, so database evolution must tolerate old and new versions during a canary. HelixWorks treats artifact rollback and database rollback as different operations and favors forward-compatible recovery.

Use three phases:

1. Expand: add nullable columns, new tables, indexes built with an availability-safe method, or dual-readable structures without removing what v1 needs.
2. Migrate: deploy code that can read the old and new shapes, write compatibly, backfill in bounded resumable batches, and measure lag and errors.
3. Contract: remove old columns, constraints, or paths only after v1 can no longer receive traffic, rollback windows expire, backups are verified, and an explicit approval is recorded.

The Day 5 v2 change adds review_status to Workboard so customers can distinguish new, reviewing, and resolved work. The expand migration adds the field with compatible behavior; v1 continues to function. v2 reads the field and writes valid values. Contract cleanup is deferred beyond the rollback window. A down migration is not assumed safe merely because a framework generated one.

A stable release at the wrong hostname or with an unrenewable certificate is not production-ready, so domain ownership and certificate lifecycle are publish gates. The customer adds a DNS record for an approved hostname, and HelixWorks verifies control before routing traffic or issuing a certificate.

Use the Automated Certificate Management Environment (ACME) protocol for issuance and renewal. Scope DNS automation narrowly when DNS validation is required; do not give an app runtime unrestricted DNS credentials. Bind the verified hostname to one customer organization and deployment, prevent cross-tenant claim races, redirect HTTP to HTTPS, enable modern TLS policy, and monitor certificate expiry and renewal failures.

Test the hostname from multiple resolvers and regions, validate the certificate chain and names, confirm Host/SNI routing reaches the intended deployment, and ensure removed domains stop routing before reassignment. Domain ownership proves control of a name; it does not grant HelixWorks project membership, app role, connector access, or tenant data.

Changing thresholds after seeing bad results turns a canary into a justification exercise, so the customer records promotion and rollback rules before traffic moves. Compare v2 with the accepted v1 baseline by customer journey, not only by process health.

Use a small internal or customer-approved cohort first, then bounded traffic steps such as 1%, 10%, 25%, 50%, and 100%. At each step require minimum sample size and observation time. Measure authenticated page success, tenant-authorized mutation success, latency, error rate, saturation, connector failures, migration/backfill health, and security invariants. A confirmed cross-tenant or authorization failure triggers immediate rollback regardless of error budget.

Rollback switches routing to the previous accepted digest and freezes expansion. It does not delete v2, reverse the expand migration, recall external side effects, or restore corrupted data. The incident record retains both artifact digests, cohorts, metrics, traces, audit events, and the exact threshold that fired.

A first release alone does not prove HelixWorks can learn safely from production, so the primary lab starts with customer feedback on v1 and completes the whole corrected-release loop. The injected v2 defect affects only one review-status path, making rollback evidence observable without inventing a database reversal.

Execute the lab in the production-like environment:

1. Click Publish for v1. Verify the deployed digest equals the tested preview digest, the production binding set is approved, the migration record is compatible, and the custom hostname presents the expected certificate.
2. Capture production feedback requesting review-status tracking and link it to revision rev-4. Build v2 once, generate provenance and SBOM, and run the acceptance suite against its digest.
3. Apply additive migration m003 for review_status while v1 serves traffic. Prove v1 reads and writes still succeed and the backfill is bounded and resumable.
4. Canary an intentionally defective v2 at 10%. Observe the predeclared mutation-error threshold breach and automatic stop; route all traffic back to v1 without reversing the schema.
5. Confirm v1 remains healthy on the expanded schema. Create corrected source revision rev-5, produce a new digest, and repeat every admission check—never relabel the failed v2 artifact.
6. Canary the corrected v2 through every step, verify tenant and authorization invariants, promote to 100%, and record customer acceptance. Leave contract cleanup blocked until the rollback window ends.

A successful route switch can hide a failed migration or mismatched artifact, so evidence must distinguish code, data, configuration, and traffic decisions. Each row records actor, resource, scope, precondition, expected and observed result, environment, timestamp, plus immutable release, artifact, migration, trace, or run identifiers.

Operators under pressure may assume one rollback button restores all state, which can worsen corruption or erase evidence. HelixWorks presents separate controls for traffic rollback, artifact selection, feature flags, credential revocation, migration pause, data restore, and external-side-effect reconciliation.

Artifact rollback is safe only while the database and bindings remain compatible with v1. Database restore is a later, destructive recovery operation with recovery-point loss and tenant implications; it is not part of routine code rollback. Connector calls, emails, webhooks, and downloaded data may require idempotent compensation or human reconciliation because routing cannot undo them. The release manifest names these limits before approval.

One Workboard release cannot prove every generated application has the same state or traffic shape, so secondary checks apply shared publish invariants to other archetypes. They remain reference coverage rather than separate labs.

For the revenue dashboard, verify production connector grants and delegated user grants are promoted or recreated explicitly, never copied from preview. For the public intake app, verify anonymous create stays bounded during canary and object scanning/queues remain compatible across versions. All three archetypes use immutable artifacts, digest-bound evidence, versioned bindings, compatible migrations where state exists, and independent artifact/data recovery.

Vendor summaries can blur artifact, certificate, rollout, and provider-adapter semantics, so delivery controls should trace unstable claims to primary specifications and official project documentation. The original sources were accessed on 2026-07-28; portable-runtime references were checked on 2026-08-06.

• SLSA Build Provenance v1.2 — build inputs, outputs, builder identity, and invocation metadata.
• SPDX 3.0 Specification — software bill of materials and supply-chain data model.
• RFC 8555: Automatic Certificate Management Environment — automated domain validation and certificate management.
• Kubernetes Deployments — declarative rollout, rollout status, revision history, and rollback mechanics.
• Daytona sandboxes — official sandbox isolation, programmatic lifecycle, and deletion-state reference for an optional build/test adapter.
• Cloudflare Sandbox SDK API — official isolated execution, file, process, session, port, tunnel, storage, and backup capability index.
• Cloudflare Sandbox tunnels — official distinction between short-lived quick tunnels and stable named-tunnel use cases.
• Cloudflare Wrangler configuration — official environment, binding, route, limit, and observability configuration reference for an optional runtime adapter.
• PostgreSQL documentation: Transaction Isolation — primary database documentation for concurrent transactional behavior; use the equivalent authoritative documentation for the selected database.

A green deployment can still serve untested bytes or unsafe state, so Publish is trustworthy only when it promotes tested bytes under validated production bindings and preserves a safe path through data change. Production feedback closes the loop only after a corrected artifact repeats admission and canary evidence.

• Build once, test by digest, and promote the same immutable artifact.
• Version production bindings and create separate workload and connector grants; never copy preview secrets or sessions.
• Expand, migrate, then contract after rollback windows—not during the first v2 canary.
• Treat artifact rollback, database recovery, and external compensation as different operations.
• Preserve failed-canary evidence and require a new digest for the corrected redeploy.
• Keep build, release, deploy, route, migrate, and verify as separate services over one DRY portable contract set.
• Treat Daytona and Cloudflare as optional capability adapters, never as implicit owners of the whole platform.

A green Publish button can conceal mismatched bytes or unsafe state changes, so each check must resolve to observed, digest-bound evidence. The failed v2 and corrected v2 are different artifacts.

• [ ] v1 production digest equals the accepted preview digest.
• [ ] Provenance signer/builder, SBOM, policy, approvals, and exceptions were verified.
• [ ] Production binding set uses production workload identity, secret references, connectors, network policy, telemetry, and scaling.
• [ ] Custom-domain ownership, TLS routing, renewal monitoring, and removal behavior were tested.
• [ ] Expand migration remained compatible with v1 and corrected v2; contract cleanup is gated.
• [ ] Predeclared canary thresholds include user journeys, data health, and immediate security invariants.
• [ ] Defective v2 triggered rollback to v1 without pretending to roll back the database.
• [ ] Corrected v2 has a new source revision and digest and repeated every admission and canary step.
• [ ] Pinned reusable IaC modules reproduced the environment, and plan/apply drift evidence names the selected provider adapter.
• [ ] Delivery-plane DR rebuilt and rerouted the accepted digest without claiming a database restore.
• [ ] Evidence rows include actor, resource, scope, precondition, expected and observed results, environment, timestamp, and immutable identifier.

Exercise the build-once promotion decision in promote-release.ts. Northstar promotes the supplier-onboarding artifact digest through environments; a rejected canary restores the previously served digest without pretending to reverse database state.

await this.deployments.deploy(command);

const decision = await this.canaries.evaluate(command);
if (!decision.approved) {
  await this.deployments.rollback({
    tenantId: command.tenantId,
    releaseId: command.releaseId,
    environment: command.environment,
    previousArtifactDigest: promotion.previousArtifactDigest,
  });
  throw new CanaryRejectedError(`Canary rejected promotion: ${decision.reason}`);
}

await this.releases.saveWithEvent(promotion.next, event);

The command declares the desired artifact digest, destination environment, and expected previous digest. The release application service interprets it, the deployment adapter changes workload routing, and the canary adapter decides whether to retain that state. On ECS, task CPU/memory and load-balancer/network capacity serve the candidate; the repository and event outbox use database disk. Deployment operations, canary observations, the rollback operation, and DeploymentPromoted.v1 are proof.

SRP separates promotion policy from deployment and canary mechanisms. DRY shares release/event contracts. IoC/DI makes both adapters replaceable. MVC limits the controller to authenticated request translation. PubSub publishes the accepted immutable fact; IaC in infra/terraform declares the ECS, queue, database, encryption, logging, and network substrate for dev, staging, and prod.

pnpm vitest run services/release-controller/src/release-controller.test.ts
pnpm smoke:product
terraform -chdir=infra/terraform validate

Configure a failed canary, supply a stale expectedPreviousArtifactDigest, skip an environment, and reuse a digest from another tenant. Evidence passes only if the candidate rolls back to the named prior digest, stale or out-of-order changes stop before promotion, and tenant release state stays isolated. This proves release control flow; production readiness additionally requires real canary signals, durable adapters, signed images, migration rehearsals, and operated rollback alerts.