Production Readiness Review for AWS

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: infra/stacks/prod/main.tf

node_instance_types = ["m7i.large"]
  node_min_size       = 3
  node_max_size       = 20
  deletion_protection = true

Code to reality

Declared intent

Launch production with multi-node baseline capacity, bounded growth, and deletion protection.

Interpreter

Terraform passes production policy to the shared environment module and the AWS provider reconciles it.

Software effect

The EKS node group maintains at least three nodes and permits controlled scaling to twenty.

Hardware effect

AWS allocates billable EC2 instances across availability zones and protects the managed database from deletion.

Observable evidence

Terraform outputs, EKS and EC2 inventories, Argo health, customer probes, and failure recovery form launch proof.

The source tables below remain the detailed contract. Begin with these customer paths:

• D30-UC-01
  • Person: Enterprise builder
  • Job: Create, generate, preview, connect, share, publish, change, operate, export, retire, and delete two different applications
  • Observable result: Both archetypes satisfy the same platform invariants while retaining distinct app-owned schemas and roles
• D30-UC-02
  • Person: Launch commander
  • Job: Admit and, if necessary, remove production traffic safely
  • Observable result: Approved digest serves customer probes, survives a bounded failure, scales, and stays within operational objectives

Now turn each customer job into a story with a result that an engineer can check:

• D30-US-01
  • Story: As an enterprise builder, I want multiple generated applications governed through the complete lifecycle, so that I can trust the platform beyond…
  • Observable acceptance: Evidence covers two archetypes, three planes, machine and human grants, positive/negative paths, exports, deletion, environment, times, runs, and…
• D30-US-02
  • Story: As a launch commander, I want explicit launch and rollback criteria, so that traffic changes follow evidence rather than optimism
  • Observable acceptance: Record names owners, criteria, observations, approvers, digest, route, SLOs, cost, failure/recovery, rollback target, decision, time, and run

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D30-FLOW-01
  • Trigger: Builder creates support-workflow and campaign-brief blueprints
  • Responsible systems: Forge control plane, generator, runtime, evidence, connector/identity services, generated apps and stores
  • Authoritative state: Provider stores for organizations/releases/grants; runtime stores for artifacts/deployments; app databases for tenants/data
  • Owned record: ProductionLaunch
  • Observable evidence: Actors, resources, scopes, expected/observed results, both app schemas, artifact/releases, grants, denials, exports/deletions, prod, times, runs,…
  • Failure signal: Cross-plane authority leak, nondeterministic artifact, failed publish/revoke/export/delete, cross-tenant access, or missing immutable evidence
• D30-FLOW-02
  • Trigger: Launch commander injects approved single-pod failure during bounded traffic
  • Responsible systems: Terraform/AWS, EKS, Argo CD, Kubernetes, admission/network, KEDA/Karpenter, observability, backup, route/probe controller
  • Authoritative state: Each owner’s desired/observed state plus immutable launch record
  • Owned record: HostedDeployment
  • Observable evidence: Old/new pod UIDs, controller/scale events, recovered response, positive control, release/rollback revision, cost, environment, time, and incident ID
  • Failure signal: Missing substrate/API, stale state/revision, unready workload, policy denial mismatch, missed objective, restore gap, or failed customer probe

Enterprise problem: “All deployments green” can conceal an unreachable API, unsafe authorization mapping, unrecoverable data, or failed customer lifecycle. Whole-course context: The prior day defines a restore dossier but the repository contains no actual AWS restore or deployment record. Today’s slice: This is a manual production-readiness review: it names the commands and evidence owners must inspect before traffic, but it is not an executable launch authorization and does not claim Zheta Forge has been deployed on AWS. End-of-day evidence: Approval, four digests, infrastructure, identity, policy, scaling, restore, positive/negative customer paths, cost, environment, times, runs, traces, and rollback target must all be real and fresh. Still unsolved: Production remains blocked until owner-approved AWS execution and an external signed governance process validate every criterion; repository validation alone is not cloud evidence.

Launch is a reversible product decision, not a deployment event. Contrasting support-workflow and campaign-brief applications falsify the claim that one worked app defines the platform limit.

No single dashboard can authorize launch because each boundary proves a different claim. These stories require a joined decision with named vetoes.

The happy path spans the complete customer lifecycle and every production layer. The recovery path validates one exact target and never broadens the blast radius.

Production launch succeeds only when substrate, cluster API, IaC state, Kubernetes runtime, GitOps revision, and customer evidence agree. A green Argo status is one boundary observation, never universal truth.

Launch evidence correlates records but does not become their owner. Provider, runtime, generated-app, and source-enterprise data retain independent retention, export, revocation, and deletion rules.

Generated-application database: Required in this slice — each candidate application must prove ownership of tenant, user, role, session, and domain records before the launch gate may pass.

A command exit is not launch evidence. The following commands collect source, account, context, release, and customer-path observations for human reviewers; their success is explicitly not launch authorization. Stable paths are scripts/verify-platform.sh, scripts/verify-release.sh, argocd/applicationsets/forge-prod.yaml, gitops/apps/forge/overlays/prod, and scripts/smoke-product.sh.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
./scripts/verify-platform.sh
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name zheta-forge-prod --alias zheta-forge-prod
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks describe-cluster --name zheta-forge-prod --query 'cluster.status'
./scripts/verify-release.sh prod
kubectl --context zheta-forge-prod get nodes
kubectl --context zheta-forge-prod -n argocd get application forge-prod -o wide
kubectl --context zheta-forge-prod -n zheta-forge rollout status deployment --all --timeout=180s
test -n "${FORGE_URL:?set FORGE_URL to the explicit reachable production control-plane URL}"
FORGE_URL="$FORGE_URL" FORGE_TOKEN="${FORGE_TOKEN:?set the explicit bearer token}" ./scripts/smoke-product.sh

If any command lacks authority, network reachability, real digests, durable Secrets, ready controllers, customer proof, or the separate restore dossier, the outcome is NO LAUNCH. Do not substitute localhost, a Kind cluster, rendered YAML, or Synced/Healthy for production evidence.

The externally governed launch dossier must contain all of the following, from one approved evidence window: source revision; Terraform version, reviewed plan, and remote-state identity; AWS profile, account, region, cluster ARN, and exact kube context; four ECR image digests and their scan/signature results; Argo Application target revision; Pod, node, and rollout identities; admission-signature and cross-tenant denials; separate workflow and dashboard archetype proofs; bounded failure and recovery; proof of the current elastic-generation launch veto; backup restore with measured recovery point and recovery time; positive and denied Forge traces using an explicit FORGE_TOKEN; budget/cost observation; and the tested rollback digest plus decision owner. A trusted system outside this repository must enforce its versioned schema, freshness window, artifact digests/signatures, distinct approver, category-specific fields, and cross-reference integrity. This lesson deliberately does not fake that trust root. If any field is missing or stale, the human decision is NO LAUNCH.

Launch does not merge the provider, hosted-runtime, and generated-application planes. Every mapping is explicit, least privilege, audited, independently revocable, and proven with both positive and denied actions.

Decision rule: if any mandatory boundary lacks fresh evidence, do not launch. Withdraw traffic before runtime deletion, export and delete generated-app data through its owner, revoke grants independently, retain immutable audit, then retire cluster and network in dependency order.

Production readiness is an evidence-backed, reversible customer decision.

• Launch only when all owners and customer probes agree.
• Two contrasting applications prove shared platform invariants without pretending their schemas are identical.
• Bounded failure, recovery, rollback, export, and deletion are launch features.

The course is complete when the learner can decide, operate, and recover without magical thinking.

• [ ] Dev, staging, and production use separate accounts, VPCs, states, roles, and EKS clusters
• [ ] One staging-proven digest reaches production through reviewed overlays and Argo CD
• [ ] Three-plane positive and denied tests pass for two application archetypes
• [ ] Immutable revision, Terraform plan/state, AWS identity/context, four digests, scans/signatures, Argo revision, and rollout identities are sealed
• [ ] Admission, tenant denial, elastic-generation veto, restore RPO/RTO, explicit-token product traces, cost, and rollback evidence are sealed