Staging EKS as the Production Rehearsal

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: infra/modules/environment/main.tf

resource "aws_subnet" "private" {
  count             = 2
  vpc_id            = aws_vpc.this.id
  availability_zone = local.azs[count.index]
  cidr_block        = cidrsubnet(var.vpc_cidr, 4, count.index + 8)
  tags = merge(local.tags, {
    "kubernetes.io/role/internal-elb" = "1"
  })
}

Code to reality

Declared intent

Place environment workloads across two private availability-zone subnets.

Interpreter

Terraform and the AWS provider create subnets and label them for internal Kubernetes load balancers.

Software effect

EKS and its controllers discover private network placement from stable subnet IDs and tags.

Hardware effect

AWS allocates isolated address ranges and routes network interfaces across two physical zones.

Observable evidence

Terraform outputs, AWS subnet inventory, EKS node addresses, and a private connectivity probe agree.

The source tables below remain the detailed contract. Begin with these customer paths:

• D23-UC-01
  • Person: Release engineer
  • Job: Rehearse the exact dev-proven digest under production-shaped topology and policy
  • Observable result: Lifecycle, migration, performance threshold, and policy probes pass in staging
• D23-UC-02
  • Person: Incident responder
  • Job: Recover a bounded staging failure before production approval
  • Observable result: One validated Forge pod failure is observed, reconciled, and customer flow recovers

Now turn each customer job into a story with a result that an engineer can check:

• D23-US-01
  • Story: As a release engineer, I want the same image digest exercised in staging, so that production approval is based on the artifact that will ship
  • Observable acceptance: Source SHA, digest, dev evidence ref, staging overlay diff, migration, positive/negative probes, environment, time, and run are recorded
• D23-US-02
  • Story: As an incident responder, I want a validated failure and recovery rehearsal, so that the runbook proves reconciliation rather than assuming it
  • Observable acceptance: Exactly one target, ownership labels, healthy precondition, blast radius, divergence, recovery, positive control, and immutable incident ID are…

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D23-FLOW-01
  • Trigger: Release engineer proposes the dev-proven digest for staging
  • Responsible systems: Terraform, EKS, Argo CD or bounded reconciler, Kubernetes controllers, Forge services, migration runner, probe runner
  • Authoritative state: Staging Terraform state, AWS APIs, Git overlay, Kubernetes API, staging app stores, Evidence Store
  • Owned record: StagingReleaseCandidate
  • Observable evidence: Actor, source, digest, dev evidence, staging cluster, migration ID, observed thresholds, environment, time, run, and traces
  • Failure signal: Wrong account, rebuilt digest, base drift, failed migration, threshold miss, policy denial mismatch, or failed lifecycle
• D23-FLOW-02
  • Trigger: Operator starts the approved single-pod failure rehearsal
  • Responsible systems: Target validator, Kubernetes Deployment controller, probe runner, rollback controller
  • Authoritative state: Kubernetes objects/events plus immutable rehearsal evidence and release records
  • Owned record: StagingTenantDataset
  • Observable evidence: Old/new pod UIDs, controller events, recovered response, positive control, rollback revision if used, staging, time, and incident ID
  • Failure signal: Zero or multiple targets, ownership mismatch, missing precondition, no replacement, affected replay failure, or failed positive control

Enterprise problem: A release that passed dev can fail under production-shaped policy, topology, or migration constraints, and calling dev “staging” hides that risk. Whole-course context: The incoming dev chapter defines evidence but does not fabricate it; today allows promotion only after real dev proof exists. Today’s slice: The public stack names zheta-forge-staging, and its overlay is intentionally blocked-unpinned until scripts/promote-release.py replaces its five image references with real ECR digests—including the inherited zero-replica broker whose cloud responsibility runs on SNS/SQS. Promotion changes immutable image references only; separate controllers and probes must establish policy, capacity, rollout, and product readiness. End-of-day evidence: Only a real isolated-account apply and rehearsal can produce the dossier. Still unsolved: Without AWS credentials, durable Secrets, digests, and connected-network access, staging remains a fail-closed design rather than a deployed environment.

Staging has value only when it can falsify a production release before customers see it. It uses synthetic or approved test data, never a casual copy of production records.

Promotion is trustworthy only when the artifact remains identical and environment differences stay explicit. These stories turn rehearsal into reviewable evidence.

Staging promotion changes a digest reference, not the artifact or credentials. The recovery path begins with a customer-visible action and ends only after both affected and unaffected probes are observed.

Staging must resemble production at the contract level while retaining separate resources and authority. Similarity means the same module, Kubernetes base, policy classes, and service responsibilities—not shared databases, clusters, or tokens.

Rehearsal evidence must not grant permission to production or copy source-owned enterprise data. Staging generated-app records use synthetic or specifically approved datasets.

Generated-application database: Required in this slice — the staging generated application owns isolated tenant and domain rows needed to prove migration, authorization, and deletion behavior.

A failure script that selects the first matching pod can damage the wrong workload, so validate a single owned target first. Run this labelled snippet from the public zheta-kubernetes-lab monorepo using stable path gitops/apps/forge/overlays/staging.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name zheta-forge-staging --alias zheta-forge-staging
kubectl --context zheta-forge-staging kustomize gitops/apps/forge/overlays/staging > /tmp/forge-staging.yaml
./scripts/verify-release.sh staging
kubectl --context zheta-forge-staging -n zheta-forge rollout status deployment --all --timeout=180s
kubectl --context zheta-forge-staging -n zheta-forge port-forward service/control-plane 18080:8080
# In another terminal: FORGE_URL=http://127.0.0.1:18080 FORGE_TOKEN="${FORGE_TOKEN:?set the explicit bearer token}" ./scripts/smoke-product.sh

The explicit context prevents a local or dev cluster from masquerading as staging; the explicit port-forward and FORGE_URL prevent the smoke script’s localhost default from masquerading as cloud proof.

A staging release role does not become a production deployer or generated-app administrator. The provider plane owns release candidates and approvals; the hosted-runtime plane owns staging artifacts, cluster, secrets, network, and deployments; generated applications own their test tenants, users, sessions, and data rules.

Decision rule: staging proves the artifact and procedure; production grants remain independently approved and revocable. Erase staging tenant data after its retention period without deleting immutable release and recovery evidence.

Staging is a falsification environment, not a waiting room for production.

• Promote the digest, not a rebuilt image.
• Match production contracts while isolating account, cluster, state, and data.
• Rehearse one bounded failure, recovery, and unaffected control.

The rehearsal is launch-worthy only when it can reject a bad candidate.

• [ ] Staging account and cluster are distinct from dev and production
• [ ] Digest matches the dev evidence exactly
• [ ] Migration, lifecycle, denial, failure, and recovery are observed
• [ ] Synthetic staging data has deletion evidence