Production Launch Gate for AWS and Ambient Mesh

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: infra/stacks/prod/main.tf

node_instance_types = ["m7i.large"]
  node_min_size       = 3
  node_max_size       = 20
  deletion_protection = true

Code to reality

Declared intent

Launch production with multi-node baseline capacity, bounded growth, and deletion protection.

Interpreter

Terraform passes production policy to the shared environment module and the AWS provider reconciles it.

Software effect

The EKS node group maintains at least three nodes and permits controlled scaling to twenty.

Hardware effect

AWS allocates billable EC2 instances across availability zones and protects the managed database from deletion.

Observable evidence

Terraform outputs, EKS and EC2 inventories, Argo health, customer probes, and failure recovery form launch proof.

The source tables below remain the detailed contract. Begin with these customer paths:

• D35-UC-01
  • Person: Enterprise builder
  • Job: Create, generate, preview, connect, share, publish, change, operate, export, retire, and delete two different applications
  • Observable result: Both archetypes satisfy the same platform invariants while retaining distinct app-owned schemas and roles
• D35-UC-02
  • Person: Launch commander
  • Job: Admit and, if necessary, remove production traffic safely
  • Observable result: Approved digest serves customer probes, survives a bounded failure, scales, and stays within operational objectives

Now turn each customer job into a story with a result that an engineer can check:

• D35-US-01
  • Story: As an enterprise builder, I want multiple generated applications governed through the complete lifecycle, so that I can trust the platform beyond…
  • Observable acceptance: Evidence covers two archetypes, three planes, machine and human grants, positive/negative paths, exports, deletion, environment, times, runs, and…
• D35-US-02
  • Story: As a launch commander, I want explicit launch and rollback criteria, so that traffic changes follow evidence rather than optimism
  • Observable acceptance: Record names owners, criteria, observations, approvers, digest, route, SLOs, cost, failure/recovery, rollback target, decision, time, and run

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D35-FLOW-01
  • Trigger: Builder creates support-workflow and campaign-brief blueprints
  • Responsible systems: Forge control plane, generator, runtime, evidence, connector/identity services, generated apps and stores
  • Authoritative state: Provider stores for organizations/releases/grants; runtime stores for artifacts/deployments; app databases for tenants/data
  • Owned record: ProductionLaunch
  • Observable evidence: Actors, resources, scopes, expected/observed results, both app schemas, artifact/releases, grants, denials, exports/deletions, prod, times, runs,…
  • Failure signal: Cross-plane authority leak, nondeterministic artifact, failed publish/revoke/export/delete, cross-tenant access, or missing immutable evidence
• D35-FLOW-02
  • Trigger: Launch commander injects approved single-pod failure during bounded traffic
  • Responsible systems: Terraform/AWS, EKS, Argo CD, Kubernetes, Istiod, ztunnel, waypoint, admission/network, observability, backup, ingress/probe controller
  • Authoritative state: Each owner’s desired/observed state plus immutable launch record
  • Owned record: HostedDeployment
  • Observable evidence: Old/new pod UIDs, controller/scale events, recovered response, positive control, release/rollback revision, cost, environment, time, and incident ID
  • Failure signal: Missing substrate/API, stale revision, unready mesh/workload, bypass, policy mismatch, probe blocked by NetworkPolicy, restore gap, or failed…

Enterprise problem: “All deployments green” can conceal an unreachable API, unsafe authorization mapping, mesh bypass, unrecoverable data, or failed customer lifecycle. Whole-course context: The incoming migration dossier defines ambient enrollment, identity policy, waypoint policy, bounded proxy recovery, and Forge-first rollback, but the repository contains no actual AWS deployment record. Today’s slice: This manual readiness review names the commands and evidence owners must inspect before traffic; it is not executable launch authorization and does not claim Forge or Istio is deployed on AWS. End-of-day evidence: Approval, five immutable image digests, EKS infrastructure, Istio/Gateway versions, mTLS, L4/L7 positive and denied paths, NetworkPolicy health-probe compatibility, ingress ownership, proxy recovery, rollback, restore, cost/latency, environment, runs, and traces must be real and fresh. Still unsolved: Production remains blocked until owner-approved AWS execution and an external signed governance process validate every criterion.

Launch is a reversible product decision, not a deployment event. Contrasting support-workflow and campaign-brief applications falsify the claim that one worked app defines the platform limit.

No single dashboard can authorize launch because each boundary proves a different claim. These stories require a joined decision with named vetoes.

The happy path spans the complete customer lifecycle and every production layer. The recovery path validates one exact target and never broadens the blast radius.

Production launch succeeds only when substrate, cluster API, IaC state, Kubernetes runtime, GitOps revision, and customer evidence agree. A green Argo status is one boundary observation, never universal truth.

Launch evidence correlates records but does not become their owner. Provider, runtime, generated-app, and source-enterprise data retain independent retention, export, revocation, and deletion rules.

Generated-application database: Required in this slice — each candidate application must prove ownership of tenant, user, role, session, and domain records before the launch gate may pass.

A command exit is not launch evidence. The following commands collect source, account, context, release, and customer-path observations for human reviewers; their success is explicitly not launch authorization. Stable paths are scripts/verify-platform.sh, scripts/verify-release.sh, argocd/applicationsets/forge-prod.yaml, gitops/apps/forge/overlays/prod, and scripts/smoke-product.sh.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
./scripts/verify-platform.sh
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name zheta-forge-prod --alias zheta-forge-prod
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks describe-cluster --name zheta-forge-prod --query 'cluster.status'
./scripts/verify-release.sh prod
MESH_CONTEXT=zheta-forge-prod ./scripts/verify-istio-ambient.sh forge
kubectl --context zheta-forge-prod get nodes
kubectl --context zheta-forge-prod -n argocd get application forge-prod -o wide
kubectl --context zheta-forge-prod -n zheta-forge rollout status deployment --all --timeout=180s
test -n "${FORGE_URL:?set FORGE_URL to the explicit reachable production control-plane URL}"
FORGE_URL="$FORGE_URL" FORGE_TOKEN="${FORGE_TOKEN:?set the explicit bearer token}" ./scripts/smoke-product.sh

If any command lacks authority, network reachability, real digests, durable Secrets, ready controllers, customer proof, or the separate restore dossier, the outcome is NO LAUNCH. Do not substitute localhost, a Kind cluster, rendered YAML, or Synced/Healthy for production evidence.

The cloud release contains five immutable image references: control plane, generator, runtime, evidence, and broker. AWS environments implement the broker responsibility with managed SNS/SQS and keep the inherited in-cluster broker Deployment at zero replicas. The broker image must still be digest-pinned: zero replicas avoids runtime Pod capacity, but it does not remove the mutable base tag from the rendered release contract.

The externally governed launch dossier must contain all of the following, from one approved evidence window: source revision; Terraform version, reviewed plan, and remote-state identity; AWS profile, account, region, cluster ARN, and exact kube context; five ECR image digests and scan/signature results, including the zero-replica broker; Argo target revision; Pod, node, and rollout identities; Istio and Gateway API versions; Istiod, CNI, and ztunnel readiness; waypoint Accepted/Programmed status; strict mTLS; positive and denied L4 principal paths; positive and denied L7 method/path traces; NetworkPolicy proof that kubelet health probes and HBONE remain compatible; ingress owner and proof of whether ingress traverses the waypoint; bounded ztunnel and waypoint recovery; Forge-first rollback rehearsal; separate workflow and dashboard archetype proofs; current elastic-generation launch veto; backup restore RPO/RTO; explicit-token product traces; mesh latency overhead and waypoint/node cost; budget observation; and tested application plus mesh rollback owners. A trusted system outside this repository must enforce schema, freshness, signatures, distinct approval, and cross-reference integrity. This lesson does not fake that trust root. If any field is missing or stale, the human decision is NO LAUNCH.

The staged local evidence separates claims. Day 32's scripts/probe-istio-l4-authorization.sh joins paired NetworkPolicy admission, reviewed evidence/generator/broker Pod UIDs, an independent kindnet denial, an HTTP GET/send/read with an any-response-byte veto, a candidate timeout/reset/broken-pipe/abort/zero-byte EOF, one destination-ztunnel record containing target IP:port plus evidence identity plus denial marker, and POST controls before/after without claiming HTTP policy. Day 33's scoped repository verifier then proves waypoint readiness, the waypoint-only L4 rewrite, POST success, and GET 403. The full-stage scripts/probe-istio-waypoint-bypass.sh can bind the same transport evidence to the waypoint state. Preserve these as source-revision evidence, but do not run the bounded probes against EKS: they deliberately veto every environment except local. A candidate transport result without the same-record ztunnel denial is inconclusive; static cloud manifests also cannot replace a separately approved live EKS denial and positive-control procedure. Until that cloud procedure exists and produces fresh evidence, the decision remains NO LAUNCH.

Launch does not merge the provider, hosted-runtime, and generated-application planes. Every mapping is explicit, least privilege, audited, independently revocable, and proven with both positive and denied actions.

Decision rule: if any mandatory boundary lacks fresh evidence, do not launch. Withdraw traffic before runtime deletion, export and delete generated-app data through its owner, revoke grants independently, retain immutable audit, unenroll Forge before shared mesh removal, then retire cluster and network in dependency order.

A production review previously stopped at infrastructure, workloads, and customer probes, leaving encrypted identity and waypoint enforcement as assumptions. The final gate now requires ambient behavior, ingress ownership, bounded proxy recovery, rollback, latency, and cost evidence from the exact EKS environment.

Production readiness is an evidence-backed, reversible customer decision.

• Launch only when all owners and customer probes agree.
• Pin all five release images; SNS/SQS owns cloud broker execution, while the inherited zero-replica broker image remains an immutable release input.
• Two contrasting applications prove shared platform invariants without pretending their schemas are identical.
• Bounded failure, recovery, rollback, export, and deletion are launch features.
• Ambient installation, enrollment, L4/L7 enforcement, proxy recovery, ingress ownership, and Forge-first rollback are independent launch gates.

The course is complete when the learner can decide, operate, and recover without magical thinking.

• [ ] Dev, staging, and production use separate accounts, VPCs, states, roles, and EKS clusters
• [ ] Five staging-proven image digests reach production through reviewed overlays and Argo CD
• [ ] Three-plane positive and denied tests pass for two application archetypes
• [ ] Immutable revision, Terraform plan/state, AWS identity/context, five digests including broker, scans/signatures, Argo revision, and rollout identities are sealed
• [ ] Local bypass artifact is sealed, and a separate approved live EKS denial/control procedure exists and passes
• [ ] Admission, tenant denial, elastic-generation veto, restore RPO/RTO, explicit-token product traces, cost, and rollback evidence are sealed