Immutable Promotion with ApplicationSet

Enterprise problem: Rebuilding an image or copying credentials during promotion makes staging evidence irrelevant and can smuggle unreviewed behavior into production. Whole-course context: The incoming bootstrap is valid only when real controller evidence exists; today gives it a fail-closed promotion contract. Today’s slice: scripts/promote-release.py accepts five real SHA-256 image digests and rewrites exactly one environment overlay; each environment-local forge-{env}.yaml ApplicationSet then reconciles only that overlay. Promotion does not add autoscaler or machine-capacity manifests. End-of-day evidence: Source SHA, five digests, one-overlay diff, approvals, Application revision, explicit FORGE_URL and FORGE_TOKEN probe, denial, time, run, and traces form one chain. Still unsolved: No promotion or AWS outcome is claimed until registries, Secrets, clusters, and approvals exist.

The customer outcome is a reviewable immutable promotion with applicationset change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.

CI release resolver, Git promotion repository, Argo CD ApplicationSet/repo/application controllers; cluster workloads; OCI registry, Git revision, Kubernetes etcd, and promotion evidence; cache: not involved in promotion authority.

This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.

Render before reconciling

Overlay mistakes are cheaper to reject before a controller changes a cluster. The real promotion command refuses placeholder registries or malformed digests and changes one exact overlay.

HelixWorks has five deployable image responsibilities: control plane, generator, runtime, evidence, and broker. Dev, staging, and production use managed SNS/SQS for the broker responsibility, so the in-cluster broker Deployment remains at zero replicas. Its base image reference is still inherited into the rendered release, however. Pinning the broker digest prevents that inherited tag from becoming an unreviewed mutable input; zero replicas is an execution choice, not permission to omit the artifact from promotion.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
for environment in local dev staging prod; do
  kubectl kustomize "gitops/apps/forge/overlays/${environment}" >/tmp/"forge-${environment}.yaml"
done
./scripts/promote-release.py staging "$ECR_REGISTRY" "$CONTROL_DIGEST" "$GENERATOR_DIGEST" "$RUNTIME_DIGEST" "$EVIDENCE_DIGEST" "$BROKER_DIGEST"
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name helixworks-forge-staging --alias helixworks-forge-staging
./scripts/verify-release.sh staging
kubectl --context helixworks-forge-staging -n argocd get application forge-staging -o wide

Provider release approval does not grant cluster shell access or generated-app roles. Hosted runtime owns artifact deployment; the generated application owns data compatibility and authorization during rollback.

Decision rule: promote immutable bytes and environment-specific evidence; retire traffic before artifact deletion, and retain release decisions after generated-app data is erased.

Promotion is an evidence chain around immutable bytes.

• ApplicationSet removes repeated wiring, not environment authority.
• Pin all five inherited images, including the zero-replica cloud broker whose responsibility is served by SNS/SQS.
• Git changes desired workload state; Argo reconciles; Kubernetes runs it.
• Rollback adds a new reviewed decision and preserves history.

The chain is complete only when bytes and evidence match.

• [ ] All cloud Applications use the approved overlays and scoped destinations
• [ ] Promotion changes digest references only
• [ ] Control-plane, generator, runtime, evidence, and broker digests are all pinned
• [ ] Production approval consumes staging evidence
• [ ] Rollback proves affected recovery and unaffected control