Staging EKS as the Production Rehearsal

Enterprise problem: A release that passed dev can fail under production-shaped policy, topology, or migration constraints, and calling dev “staging” hides that risk. Whole-course context: The incoming dev chapter defines evidence but does not fabricate it; today allows promotion only after real dev proof exists. Today’s slice: The public stack names helixworks-forge-staging, and its overlay is intentionally blocked-unpinned until scripts/promote-release.py replaces its five image references with real ECR digests—including the inherited zero-replica broker whose cloud responsibility runs on SNS/SQS. Promotion changes immutable image references only; separate controllers and probes must establish policy, capacity, rollout, and product readiness. End-of-day evidence: Only a real isolated-account apply and rehearsal can produce the dossier. Still unsolved: Without AWS credentials, durable Secrets, digests, and connected-network access, staging remains a fail-closed design rather than a deployed environment.

The customer outcome is a reviewable staging eks as the production rehearsal change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.

Staging EKS control plane, Argo CD, CI promotion gate, and backup/restore controller; account-local nodes/pods; staging state backend, etcd snapshots, registry, and evidence objects; cache: not involved.

This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.

Run the bounded rehearsal

A failure script that selects the first matching pod can damage the wrong workload, so validate a single owned target first. Run this labelled snippet from the public helixworks-kubernetes-lab monorepo using stable path gitops/apps/forge/overlays/staging.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name helixworks-forge-staging --alias helixworks-forge-staging
kubectl --context helixworks-forge-staging kustomize gitops/apps/forge/overlays/staging > /tmp/forge-staging.yaml
./scripts/verify-release.sh staging
kubectl --context helixworks-forge-staging -n helixworks-forge rollout status deployment --all --timeout=180s
kubectl --context helixworks-forge-staging -n helixworks-forge port-forward service/control-plane 18080:8080
# In another terminal: FORGE_URL=http://127.0.0.1:18080 FORGE_TOKEN="${FORGE_TOKEN:?set the explicit bearer token}" ./scripts/smoke-product.sh

The explicit context prevents a local or dev cluster from masquerading as staging; the explicit port-forward and FORGE_URL prevent the smoke script’s localhost default from masquerading as cloud proof.

A staging release role does not become a production deployer or generated-app administrator. The provider plane owns release candidates and approvals; the hosted-runtime plane owns staging artifacts, cluster, secrets, network, and deployments; generated applications own their test tenants, users, sessions, and data rules.

Decision rule: staging proves the artifact and procedure; production grants remain independently approved and revocable. Erase staging tenant data after its retention period without deleting immutable release and recovery evidence.

Staging is a falsification environment, not a waiting room for production.

• Promote the digest, not a rebuilt image.
• Match production contracts while isolating account, cluster, state, and data.
• Rehearse one bounded failure, recovery, and unaffected control.

The rehearsal is launch-worthy only when it can reject a bad candidate.

• [ ] Staging account and cluster are distinct from dev and production
• [ ] Digest matches the dev evidence exactly
• [ ] Migration, lifecycle, denial, failure, and recovery are observed
• [ ] Synthetic staging data has deletion evidence