Secure Delivery, Promotion, and Rollback

Enterprise problem: A customer can test a preview successfully yet publish different bytes, bindings, or database assumptions, turning “Publish” into an unreviewed rebuild with no safe recovery path.

Whole-course context: Days 03–04 produced a revision-bound stateful preview and governed connectors, while the companion authorization course defines separate identities and sharing/revocation evidence; today promotes those exact tested inputs rather than recreating them.

Today’s slice: The customer releases v1, attaches a stable custom domain, responds to production feedback with a backward-compatible v2 migration, canaries v2, rolls back on a threshold breach, fixes it, and redeploys.

End-of-day evidence: Digest-bound v1/v2 manifests, provenance, SBOM, migration records, domain/TLS checks, canary observations, rollback traces, and corrected-redeploy results all carry immutable artifact or run IDs.

Still unsolved: Cross-fleet SLO ownership, incidents, break-glass, backup restoration, residency, retention, export, deletion, and retirement remain deferred.

A release course can become a catalogue of delivery tools while leaving the customer's publish and recovery jobs implicit, so operators may optimize individual gates without proving a complete application change is controllable. These use cases bind each product action to visible success and failure evidence.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D05-UC-01	Workboard app owner	Publish the verified v1 application through one accountable action	rel-workboard-v1 admits the Day 3 BuildArtifact, creates dep-workboard-v1 in the reused production AppEnvironment, and serves the approved digest	Missing, stale, or mismatched attestation evidence denies admission before production routing changes
D05-UC-02	Customer domain administrator	Attach a verified custom domain and managed TLS	Ownership verifies, one active DomainBinding routes to dep-workboard-v1, and a browser observes a valid certificate and Workboard response	Failed ownership or certificate verification leaves the prior platform route active and records the denial
D05-UC-03	Release operator	Add todo review_status, apply a compatible migration, and canary v2	Candidate 1 remains immutable, canary evidence detects a blocking journey failure, and traffic returns to v1 while the expanded schema remains	Recovery proves v1 service without routine database restore or destructive schema reversal
D05-UC-04	App owner and release approver	Correct the v2 defect and redeploy without overwriting failed evidence	New source, digest, rel-workboard-v2-candidate-2, and dep-workboard-v2-candidate-2 repeat admission and canary gates	Reusing a failed release, deployment, revision, or digest is denied as an identity violation

Release machinery without actor-centred acceptance can report green while the owner or operator still cannot complete the intended job, so every use case needs an observable story rather than a feature wish.

Story ID	Use case IDs	User story	Observable acceptance conditions
D05-US-01	D05-UC-01	As a Workboard app owner, I want one Publish action to bind approved evidence to exact bytes, so that I know which v1 artifact customers receive.	The product shows release, digest, source revision, approver, environment, and deployment; a mismatch produces a visible hold with no route change.
D05-US-02	D05-UC-02	As a customer domain administrator, I want to prove domain ownership and activate managed TLS, so that users reach Workboard at an approved address.	Challenge, certificate status, active binding, DNS target, TLS probe, and prior-route behavior are observable.
D05-US-03	D05-UC-03	As a release operator, I want to canary the review_status change and route back safely on failure, so that customer data remains usable while exposure is bounded.	The schema accepts v1 and v2, declared thresholds halt candidate 1, v1 journeys recover, and no routine data restore or contraction occurs.
D05-US-04	D05-UC-04	As a release approver, I want corrected v2 to have new immutable identities and repeat every gate, so that failed evidence cannot be rewritten into a pass.	Candidate 2 has a distinct revision, digest, release, and deployment; admission and canary evidence binds them before promotion.

A publish button is unsafe when hidden steps can swap bytes, partially bind a domain, or confuse routing rollback with data rollback, so these flows start with a customer-visible action and end in inspectable evidence.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D05-FLOW-01	D05-UC-01	Happy	App owner selects Publish v1	1. Resolve the approved Day 3 artifact and production environment; 2. verify provenance, signature, SBOM, policy, and approvers; 3. create immutable release and deployment; 4. route and run tenant-denial journeys	Receipt names rel-workboard-v1, dep-workboard-v1, digest, environment, decision, route, and probes
D05-FLOW-02	D05-UC-02	Denied	Domain administrator submits a hostname without valid ownership proof	1. Create a pending binding; 2. issue the ownership challenge; 3. deny certificate and route activation; 4. verify the platform hostname still serves v1	Binding denial, challenge observation, no certificate, and unchanged-route probe
D05-FLOW-03	D05-UC-03	Failure	Operator starts candidate 1 canary after the compatible review_status expansion	1. Admit immutable candidate 1; 2. apply the additive migration while v1 remains compatible; 3. route a bounded cohort; 4. observe the declared journey breach; 5. halt expansion	Failed observation binds rel-workboard-v2-candidate-1 and dep-workboard-v2-candidate-1; promotion never occurs
D05-FLOW-04	D05-UC-03	Recovery	Operator selects Route back to v1	1. Shift candidate traffic to dep-workboard-v1; 2. verify v1 user and denial journeys; 3. retain expanded schema and migration history; 4. record that restore and contract migration were not invoked	Rollback event shows recovered v1 routing, healthy journeys, compatible schema, and no routine data restore
D05-FLOW-05	D05-UC-04	Happy	App owner submits corrected v2 source	1. Build a new digest; 2. create candidate 2 release and deployment; 3. repeat admission and migration-compatibility gates; 4. canary for the full window; 5. promote	Receipt binds corrected source and digest to rel-workboard-v2-candidate-2 and dep-workboard-v2-candidate-2; candidate 1 remains immutable

An architecture list not derived from publish and recovery steps can omit the component that owns a denial or rollback, so each use case maps to its entry point, responsible services, authoritative state, and failure evidence.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D05-UC-01	Publish v1 action	Release service, artifact registry, provenance and SBOM verifier, admission controller, environment binding service, deployment controller, traffic manager	Reused Day 3 artifact registry and environment controller plus append-only release and deployment stores	Admission decision names missing or mismatched evidence; deployment and route remain absent or unchanged
D05-UC-02	Add custom domain action	Domain-binding service, ownership verifier, DNS controller, certificate manager, gateway	Domain-binding store owns pending and active bindings; DNS and certificate providers execute changes	Challenge observation, certificate failure, uniqueness conflict, and unchanged-route probe
D05-UC-03	Start v2 canary and Route back actions	Build service, migration controller, admission controller, deployment controller, traffic manager, journey and data-health probes, rollback controller	Artifact, migration, release, deployment, observation, decision, rollback stores; Workboard database owns review_status	Failed threshold binds candidate 1; rollback proves v1 routing and compatible schema without restore
D05-UC-04	Submit corrected v2 and Promote action	Source-revision service, build service, evidence verifiers, admission controller, deployment controller, traffic manager	Immutable source, artifact, release, deployment, observation, and decision records	Identity-reuse denial or repeated gate failure holds candidate 2 without altering candidate 1

Release state becomes untrustworthy when a new table duplicates an existing artifact or environment authority, and rollback becomes destructive when deployment state is confused with customer data. This model reuses Day 3 ownership and records artifact, routing, migration, and data lifecycles separately.

Generated-application database: Required in this slice — the Workboard data service owns tenant-scoped todos and the additive review_status field exercised by v1, failed v2, routing rollback, and corrected v2.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
BuildArtifact	Reused Day 3 artifact registry owned by build service	build_artifact_digest	source_revision references immutable source	organization_id partitions customer artifacts	Digest and bytes are immutable and cannot be rebound	Retained through evidence and rollback windows, then expired by artifact policy	D05-UC-01, D05-UC-03, D05-UC-04
AppEnvironment	Reused Day 3 environment store owned by environment controller	environment_id	application_id references app-workboard	organization_id partitions environments	Production identity, region, network, secret policy, and database binding are explicit	Retained while deployed; destroyed after domain, data, and evidence lifecycle checks	D05-UC-01, D05-UC-02, D05-UC-03, D05-UC-04
Attestation	Provenance store owned by supply-chain verifier	attestation_id	build_artifact_digest is an immutable artifact reference	organization_id scopes policy	Subject digest, builder, source, build type, and parameters must match policy	Retained append-only with release evidence, then expired under governed retention	D05-UC-01, D05-UC-03, D05-UC-04
Release	Release store owned by release service	release_id	build_artifact_digest and environment_id bind bytes and target	organization_id partitions releases	v1, failed candidate 1, and corrected candidate 2 are immutable distinct IDs	Retained as release history; never overwritten, then archived or expired by policy	D05-UC-01, D05-UC-03, D05-UC-04
Deployment	Deployment store owned by deployment controller	deployment_id	release_id and environment_id bind release to runtime	organization_id partitions deployments	Idempotency key is unique per target and release; all three deployment IDs remain distinct	Superseded deployments scale down but remain through rollback and investigation before purge	D05-UC-01, D05-UC-03, D05-UC-04
DomainBinding	Domain store owned by domain-binding service	domain_binding_id	environment_id references production	organization_id partitions domain ownership	One active binding per hostname; ownership and certificate verify before routing	Pending and denied bindings expire; active binding is removed with DNS and TLS evidence	D05-UC-02
MigrationPlan	Migration store owned by migration controller	migration_plan_id	application_id and target source_revision reference the change	organization_id partitions migration authority	State follows expand, migrate, contract; expansion remains compatible with v1 and v2	Retained with schema history; contract cleanup waits beyond rollback and never accompanies route rollback	D05-UC-03, D05-UC-04
MigrationExecution	Migration store owned by migration controller	migration_execution_id	migration_plan_id, environment_id, and deployment reference bind execution	organization_id partitions execution	Checksum and idempotency key prevent divergent or repeated application	Retained as evidence; compatible expansion remains after route rollback	D05-UC-03, D05-UC-04
CanaryObservation	Observation store owned by rollout controller	canary_observation_id	deployment_id references candidate and baseline	organization_id partitions observations	Window, cohort, journey, threshold, value, and digest are immutable	Retained append-only through release and incident windows, then expires by policy	D05-UC-03, D05-UC-04
RolloutDecision	Decision store owned by release authority	rollout_decision_id	release_id references exact candidate	organization_id partitions decisions	Promote, halt, or route-back includes actor, evidence, and timestamp	Retained append-only; superseded decisions expire only under evidence policy	D05-UC-03, D05-UC-04
RollbackEvent	Rollback store owned by rollback controller	rollback_event_id	Failed and target deployment_id values identify routing change	organization_id partitions rollback	Routing, artifact, configuration, and data actions are separate; routine data restore is false	Retained append-only with recovery evidence; never rewritten into success	D05-UC-03
Todo	Workboard database owned by Workboard data service	Compound app_tenant_id, todo_id	Compound app_tenant_id, board_id references same-tenant board	app_tenant_id is the app tenant boundary	review_status is backward-compatible for v1 and constrained to declared v2 states	Retained under app policy; deleted with tenant or app after export and holds	D05-UC-03, D05-UC-04

If Publish compiles mutable source again, the production result can differ from the preview the customer accepted. HelixWorks therefore builds once in an isolated builder, identifies the output by cryptographic digest, tests that digest, and promotes the same immutable artifact through environments.

Environment-specific configuration remains outside the artifact but is versioned in a release manifest. Promotion verifies the digest, signature, provenance predicate, software bill of materials (SBOM), acceptance evidence, policy, and approvers before creating a deployment. It never trusts a tag such as latest or a filename as identity.

A dashboard can show green checks from another revision or environment, so the admission decision needs one immutable join key. The release manifest binds source, build, artifact, tests, data changes, runtime bindings, policy, and customer approval.

{
  "releaseId": "rel-workboard-v1",
  "sourceRevision": "rev-3",
  "artifactDigest": "sha256:8a94...c12e",
  "buildRunId": "build-07-001",
  "provenanceId": "att-07-001",
  "sbomDigest": "sha256:26de...91aa",
  "acceptanceRunId": "accept-06-014",
  "migrationSet": ["m001", "m002"],
  "bindingSet": "bindings-prod-17",
  "policyVersion": "release-policy-9",
  "decision": "release"
}

The admission controller verifies that evidence was observed against the same digest or an explicitly declared pre-deployment condition. It rejects stale scans, unsigned or unexpected provenance, an unapproved builder, unresolved critical findings, missing migration compatibility proof, expired exceptions, and mutable dependencies. A human release, release-with-time-bounded-exception, or hold decision remains attributable and scoped.

An exact artifact can still behave incorrectly if production silently inherits preview databases, tokens, or networks. Runtime bindings are environment-owned references to secrets, databases, object storage, workload identity, connector grants, egress policy, telemetry, scaling, and domains; they are validated separately from artifact identity.

The production binding set names secret references and versions, never secret values. A deployment receives a new production workload identity and can request only production-scoped credentials. Day 4’s connector definition may be promoted by an administrator, but preview and production grants remain distinct; the companion authorization course keeps user delegation user- and source-scoped. The publish action does not copy refresh tokens, preview data, customer source data, or generated-app sessions.

Run preflight checks from the production network and identity: resolve required endpoints, authenticate through the broker, execute a harmless authorized source probe, deny unapproved destinations, verify database role and tenant predicates, and confirm telemetry intake. Network reachability is not authorization, and successful secret resolution is not permission to use every resource behind that secret.

Rolling back code while a destructive schema change remains can make the prior artifact fail, so database evolution must tolerate old and new versions during a canary. HelixWorks treats artifact rollback and database rollback as different operations and favors forward-compatible recovery.

Use three phases:

1. Expand: add nullable columns, new tables, indexes built with an availability-safe method, or dual-readable structures without removing what v1 needs.
2. Migrate: deploy code that can read the old and new shapes, write compatibly, backfill in bounded resumable batches, and measure lag and errors.
3. Contract: remove old columns, constraints, or paths only after v1 can no longer receive traffic, rollback windows expire, backups are verified, and an explicit approval is recorded.

The Day 5 v2 change adds feedback_status to Workboard so customers can distinguish new, reviewing, and resolved feedback. The expand migration adds the field with compatible behavior; v1 continues to function. v2 reads the field and writes valid values. Contract cleanup is deferred beyond the rollback window. A down migration is not assumed safe merely because a framework generated one.

A stable release at the wrong hostname or with an unrenewable certificate is not production-ready, so domain ownership and certificate lifecycle are publish gates. The customer adds a DNS record for an approved hostname, and HelixWorks verifies control before routing traffic or issuing a certificate.

Use the Automated Certificate Management Environment (ACME) protocol for issuance and renewal. Scope DNS automation narrowly when DNS validation is required; do not give an app runtime unrestricted DNS credentials. Bind the verified hostname to one customer organization and deployment, prevent cross-tenant claim races, redirect HTTP to HTTPS, enable modern TLS policy, and monitor certificate expiry and renewal failures.

Test the hostname from multiple resolvers and regions, validate the certificate chain and names, confirm Host/SNI routing reaches the intended deployment, and ensure removed domains stop routing before reassignment. Domain ownership proves control of a name; it does not grant HelixWorks project membership, app role, connector access, or tenant data.

Changing thresholds after seeing bad results turns a canary into a justification exercise, so the customer records promotion and rollback rules before traffic moves. Compare v2 with the accepted v1 baseline by customer journey, not only by process health.

Use a small internal or customer-approved cohort first, then bounded traffic steps such as 1%, 10%, 25%, 50%, and 100%. At each step require minimum sample size and observation time. Measure authenticated page success, tenant-authorized mutation success, latency, error rate, saturation, connector failures, migration/backfill health, and security invariants. A confirmed cross-tenant or authorization failure triggers immediate rollback regardless of error budget.

Rollback switches routing to the previous accepted digest and freezes expansion. It does not delete v2, reverse the expand migration, recall external side effects, or restore corrupted data. The incident record retains both artifact digests, cohorts, metrics, traces, audit events, and the exact threshold that fired.

A first release alone does not prove HelixWorks can learn safely from production, so the primary lab starts with customer feedback on v1 and completes the whole corrected-release loop. The injected v2 defect affects only one feedback-status path, making rollback evidence observable without inventing a database reversal.

Execute the lab in the production-like environment:

1. Click Publish for v1. Verify the deployed digest equals the tested preview digest, the production binding set is approved, the migration record is compatible, and the custom hostname presents the expected certificate.
2. Capture production feedback requesting feedback-status tracking and link it to revision rev-4. Build v2 once, generate provenance and SBOM, and run the acceptance suite against its digest.
3. Apply additive migration m003 for feedback_status while v1 serves traffic. Prove v1 reads and writes still succeed and the backfill is bounded and resumable.
4. Canary an intentionally defective v2 at 10%. Observe the predeclared mutation-error threshold breach and automatic stop; route all traffic back to v1 without reversing the schema.
5. Confirm v1 remains healthy on the expanded schema. Create corrected source revision rev-5, produce a new digest, and repeat every admission check—never relabel the failed v2 artifact.
6. Canary the corrected v2 through every step, verify tenant and authorization invariants, promote to 100%, and record customer acceptance. Leave contract cleanup blocked until the rollback window ends.

A successful route switch can hide a failed migration or mismatched artifact, so evidence must distinguish code, data, configuration, and traffic decisions. Each row records actor, resource, scope, precondition, expected and observed result, environment, timestamp, plus immutable release, artifact, migration, trace, or run identifiers.

Claim	Required observation	Immutable join
Exact v1 published	Registry digest, runtime digest, and tested digest equal	rel-workboard-v1, artifact digest, acceptance run
Production bindings valid	Correct workload, secret refs, connector scope, network denials	Binding-set version and preflight trace IDs
Migration compatible	v1 and v2 operate on expanded schema; backfill metrics bounded	Migration run and schema version
Canary rollback works	Threshold fires; v2 receives no new traffic; v1 recovers	Routing event, threshold snapshot, both digests
Corrected redeploy works	New digest passes admission and every canary step	New build, provenance, SBOM, acceptance and canary run IDs
Domain/TLS works	DNS control, certificate, routing, renewal monitor	Domain verification and certificate IDs

Preserve the failed v2 evidence; failure is part of the proof. A rollback rehearsal that omits the corrected redeploy tests escape, not recovery.

Operators under pressure may assume one rollback button restores all state, which can worsen corruption or erase evidence. HelixWorks presents separate controls for traffic rollback, artifact selection, feature flags, credential revocation, migration pause, data restore, and external-side-effect reconciliation.

Artifact rollback is safe only while the database and bindings remain compatible with v1. Database restore is a later, destructive recovery operation with recovery-point loss and tenant implications; it is not part of routine code rollback. Connector calls, emails, webhooks, and downloaded data may require idempotent compensation or human reconciliation because routing cannot undo them. The release manifest names these limits before approval.

One Workboard release cannot prove every generated application has the same state or traffic shape, so secondary checks apply shared publish invariants to other archetypes. They remain reference coverage rather than separate labs.

For the revenue dashboard, verify production connector grants and delegated user grants are promoted or recreated explicitly, never copied from preview. For the public intake app, verify anonymous create stays bounded during canary and object scanning/queues remain compatible across versions. All three archetypes use immutable artifacts, digest-bound evidence, versioned bindings, compatible migrations where state exists, and independent artifact/data recovery.

Delivery controls are strongest when their artifact, certificate, and rollout semantics come from primary specifications and project documentation. These sources were accessed on 2026-07-28.

• SLSA Build Provenance v1.2 — build inputs, outputs, builder identity, and invocation metadata.
• SPDX 3.0 Specification — software bill of materials and supply-chain data model.
• RFC 8555: Automatic Certificate Management Environment — automated domain validation and certificate management.
• Kubernetes Deployments — declarative rollout, rollout status, revision history, and rollback mechanics.
• PostgreSQL documentation: Transaction Isolation — primary database documentation for concurrent transactional behavior; use the equivalent authoritative documentation for the selected database.

Publish is trustworthy only when it promotes tested bytes under validated production bindings and preserves a safe path through data change. Production feedback closes the loop only after a corrected artifact repeats admission and canary evidence.

• Build once, test by digest, and promote the same immutable artifact.
• Version production bindings and create separate workload and connector grants; never copy preview secrets or sessions.
• Expand, migrate, then contract after rollback windows—not during the first v2 canary.
• Treat artifact rollback, database recovery, and external compensation as different operations.
• Preserve failed-canary evidence and require a new digest for the corrected redeploy.

A green Publish button can conceal mismatched bytes or unsafe state changes, so each check must resolve to observed, digest-bound evidence. The failed v2 and corrected v2 are different artifacts.

• [ ] v1 production digest equals the accepted preview digest.
• [ ] Provenance signer/builder, SBOM, policy, approvals, and exceptions were verified.
• [ ] Production binding set uses production workload identity, secret references, connectors, network policy, telemetry, and scaling.
• [ ] Custom-domain ownership, TLS routing, renewal monitoring, and removal behavior were tested.
• [ ] Expand migration remained compatible with v1 and corrected v2; contract cleanup is gated.
• [ ] Predeclared canary thresholds include user journeys, data health, and immediate security invariants.
• [ ] Defective v2 triggered rollback to v1 without pretending to roll back the database.
• [ ] Corrected v2 has a new source revision and digest and repeated every admission and canary step.
• [ ] Evidence rows include actor, resource, scope, precondition, expected and observed results, environment, timestamp, and immutable identifier.