Immutable Promotion with ApplicationSet

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: scripts/promote-release.py

if environment not in {"dev", "staging", "prod"} or len(digests) != 4:
    raise SystemExit("usage: promote-release.py dev|staging|prod ECR_REGISTRY CONTROL GENERATOR RUNTIME EVIDENCE")
if not re.fullmatch(r"[0-9]{12}\.dkr\.ecr\.[a-z0-9-]+\.amazonaws\.com", registry):
    raise SystemExit("registry must be a real account-scoped ECR registry")
if any(not re.fullmatch(r"sha256:[0-9a-f]{64}", digest) for digest in digests):
    raise SystemExit("every service must have a real sha256 digest")

Code to reality

Declared intent

Promote exactly one environment only when all four service images have real account-scoped digests.

Interpreter

Python validates the environment, ECR registry, and immutable digest syntax before rewriting one overlay.

Software effect

Promotion changes one environment repository artifact; each environment-local Argo instance remains independent.

Hardware effect

Only the selected cluster may later replace Pods with the approved image bytes.

Observable evidence

One-overlay diff, four ECR digests, review approval, and the matching local Argo revision prove promotion.

The source tables below remain the detailed contract. Begin with these customer paths:

• D26-UC-01
  • Person: Release manager
  • Job: Promote one dev-proven digest through staging to production
  • Observable result: Each overlay references the same digest after its own approval and probes
• D26-UC-02
  • Person: On-call engineer
  • Job: Roll back a harmful release without losing audit history
  • Observable result: Previous proven digest reconciles and customer flow recovers

Now turn each customer job into a story with a result that an engineer can check:

• D26-US-01
  • Story: As a release manager, I want digest-only promotion with per-environment approval, so that evidence follows identical bytes
  • Observable acceptance: Chain contains source, digest, dev/staging proofs, overlay diffs, approvers, Argo revisions, responses, times, and run IDs
• D26-US-02
  • Story: As an on-call engineer, I want rollback to a previously proven digest, so that service recovers without deleting the failed release record
  • Observable acceptance: Old/new digest, reason, approval, reconciliation, affected replay, positive control, time, run, and traces are recorded

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D26-FLOW-01
  • Trigger: Release manager promotes a dev-proven digest
  • Responsible systems: Git review, registry, ApplicationSet, Argo application controller, Kubernetes rollout, Forge probe
  • Authoritative state: Git and registry for artifact intent; Kubernetes APIs for rollout; release/evidence stores for approval and result
  • Owned record: ArtifactAttestation
  • Observable evidence: Source, digest, diffs, approvals, cluster/Application revisions, customer responses, environment, time, run, and traces
  • Failure signal: Mutable tag, digest mismatch, wrong overlay/cluster, missing approval, degraded rollout, or failed response
• D26-FLOW-02
  • Trigger: Production probe breaches release acceptance
  • Responsible systems: Release controller, Argo CD, Kubernetes Deployment, probe runner
  • Authoritative state: Immutable release history plus Git overlay revisions and Application status
  • Owned record: PromotionDecision
  • Observable evidence: Failed and restored digests, commits, sync/health, recovered response, positive control, actor, time, and incident ID
  • Failure signal: Unproven target, direct live mutation self-healed, rollback timeout, or positive-control failure

Enterprise problem: Rebuilding an image or copying credentials during promotion makes staging evidence irrelevant and can smuggle unreviewed behavior into production. Whole-course context: The incoming bootstrap is valid only when real controller evidence exists; today gives it a fail-closed promotion contract. Today’s slice: scripts/promote-release.py accepts four real SHA-256 image digests and rewrites exactly one environment overlay; each environment-local forge-{env}.yaml ApplicationSet then reconciles only that overlay. Promotion does not add autoscaler or machine-capacity manifests. End-of-day evidence: Source SHA, four digests, one-overlay diff, approvals, Application revision, explicit FORGE_URL and FORGE_TOKEN probe, denial, time, run, and traces form one chain. Still unsolved: No promotion or AWS outcome is claimed until registries, Secrets, clusters, and approvals exist.

An environment name is not a release, so promotion must identify the exact artifact and evidence that justify it. Rollback selects a previous digest rather than rewriting history.

Promotion policy fails when the artifact and environment configuration are mixed. These stories keep the reusable base stable and overlays small.

The customer-visible trigger is a release decision and the terminal event is a verified product response. Argo CD is one controller in that path, not the evidence itself.

ApplicationSet removes repeated Application fields, but it must not collapse environment approval or credentials. Each environment-local Argo instance renders its own manifest and targets the fixed https://kubernetes.default.svc destination with its in-cluster identity.

A release record must survive rollback, or recovery destroys the audit trail. Existing generated-app data remains with each application and is not copied during deployment.

Generated-application database: Required in this slice — each generated application keeps its tenant data through release and rollback; promotion changes runtime artifacts, not data ownership.

Overlay mistakes are cheaper to reject before a controller changes a cluster. The real promotion command refuses placeholder registries or malformed digests and changes one exact overlay.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
for environment in local dev staging prod; do
  kubectl kustomize "gitops/apps/forge/overlays/${environment}" >/tmp/"forge-${environment}.yaml"
done
./scripts/promote-release.py staging "$ECR_REGISTRY" "$CONTROL_DIGEST" "$GENERATOR_DIGEST" "$RUNTIME_DIGEST" "$EVIDENCE_DIGEST"
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name zheta-forge-staging --alias zheta-forge-staging
./scripts/verify-release.sh staging
kubectl --context zheta-forge-staging -n argocd get application forge-staging -o wide

Provider release approval does not grant cluster shell access or generated-app roles. Hosted runtime owns artifact deployment; the generated application owns data compatibility and authorization during rollback.

Decision rule: promote immutable bytes and environment-specific evidence; retire traffic before artifact deletion, and retain release decisions after generated-app data is erased.

Promotion is an evidence chain around immutable bytes.

• ApplicationSet removes repeated wiring, not environment authority.
• Git changes desired workload state; Argo reconciles; Kubernetes runs it.
• Rollback adds a new reviewed decision and preserves history.

The chain is complete only when bytes and evidence match.

• [ ] All cloud Applications use the approved overlays and scoped destinations
• [ ] Promotion changes digest references only
• [ ] Production approval consumes staging evidence
• [ ] Rollback proves affected recovery and unaffected control