Production Launch Gate for AWS and Ambient Mesh

Enterprise problem: “All deployments green” can conceal an unreachable API, unsafe authorization mapping, mesh bypass, unrecoverable data, or failed customer lifecycle. Whole-course context: The incoming migration dossier defines ambient enrollment, identity policy, waypoint policy, bounded proxy recovery, and HelixWorks-first rollback, but the repository contains no actual AWS deployment record. Today’s slice: This manual readiness review names the commands and evidence owners must inspect before traffic; it is not executable launch authorization and does not claim HelixWorks or Istio is deployed on AWS. End-of-day evidence: Approval, five immutable image digests, EKS infrastructure, Istio/Gateway versions, mTLS, L4/L7 positive and denied paths, NetworkPolicy health-probe compatibility, ingress ownership, proxy recovery, rollback, restore, cost/latency, environment, runs, and traces must be real and fresh. Still unsolved: Production remains blocked until owner-approved AWS execution and an external signed governance process validate every criterion.

The customer outcome is a reviewable production launch gate for aws and ambient mesh change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.

Terraform/AWS, EKS, Argo CD, ingress/DNS/TLS, identity, observability, and backup services; multi-AZ compute/network; state backend, Git, registry, databases, backups, and immutable launch evidence; cache: only application-owned and independently recoverable.

This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.

Collect evidence for the manual readiness review

A command exit is not launch evidence. The following commands collect source, account, context, release, and customer-path observations for human reviewers; their success is explicitly not launch authorization. Stable paths are scripts/verify-platform.sh, scripts/verify-release.sh, argocd/applicationsets/forge-prod.yaml, gitops/apps/forge/overlays/prod, and scripts/smoke-product.sh.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
./scripts/verify-platform.sh
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name helixworks-forge-prod --alias helixworks-forge-prod
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks describe-cluster --name helixworks-forge-prod --query 'cluster.status'
./scripts/verify-release.sh prod
MESH_CONTEXT=helixworks-forge-prod ./scripts/verify-istio-ambient.sh forge
kubectl --context helixworks-forge-prod get nodes
kubectl --context helixworks-forge-prod -n argocd get application forge-prod -o wide
kubectl --context helixworks-forge-prod -n helixworks-forge rollout status deployment --all --timeout=180s
test -n "${FORGE_URL:?set FORGE_URL to the explicit reachable production control-plane URL}"
FORGE_URL="$FORGE_URL" FORGE_TOKEN="${FORGE_TOKEN:?set the explicit bearer token}" ./scripts/smoke-product.sh

If any command lacks authority, network reachability, real digests, durable Secrets, ready controllers, customer proof, or the separate restore dossier, the outcome is NO LAUNCH. Do not substitute localhost, a Kind cluster, rendered YAML, or Synced/Healthy for production evidence.

The cloud release contains five immutable image references: control plane, generator, runtime, evidence, and broker. AWS environments implement the broker responsibility with managed SNS/SQS and keep the inherited in-cluster broker Deployment at zero replicas. The broker image must still be digest-pinned: zero replicas avoids runtime Pod capacity, but it does not remove the mutable base tag from the rendered release contract.

The externally governed launch dossier must contain all of the following, from one approved evidence window: source revision; Terraform version, reviewed plan, and remote-state identity; AWS profile, account, region, cluster ARN, and exact kube context; five ECR image digests and scan/signature results, including the zero-replica broker; Argo target revision; Pod, node, and rollout identities; Istio and Gateway API versions; Istiod, CNI, and ztunnel readiness; waypoint Accepted/Programmed status; strict mTLS; positive and denied L4 principal paths; positive and denied L7 method/path traces; NetworkPolicy proof that kubelet health probes and HBONE remain compatible; ingress owner and proof of whether ingress traverses the waypoint; bounded ztunnel and waypoint recovery; HelixWorks-first rollback rehearsal; separate workflow and dashboard archetype proofs; current elastic-generation launch veto; backup restore RPO/RTO; explicit-token product traces; mesh latency overhead and waypoint/node cost; budget observation; and tested application plus mesh rollback owners. A trusted system outside this repository must enforce schema, freshness, signatures, distinct approval, and cross-reference integrity. This lesson does not fake that trust root. If any field is missing or stale, the human decision is NO LAUNCH.

The staged local evidence separates claims. Day 32's scripts/probe-istio-l4-authorization.sh joins paired NetworkPolicy admission, reviewed evidence/generator/broker Pod UIDs, an independent kindnet denial, an HTTP GET/send/read with an any-response-byte veto, a candidate timeout/reset/broken-pipe/abort/zero-byte EOF, one destination-ztunnel record containing target IP:port plus evidence identity plus denial marker, and POST controls before/after without claiming HTTP policy. Day 33's scoped repository verifier then proves waypoint readiness, the waypoint-only L4 rewrite, POST success, and GET 403. The full-stage scripts/probe-istio-waypoint-bypass.sh can bind the same transport evidence to the waypoint state. Preserve these as source-revision evidence, but do not run the bounded probes against EKS: they deliberately veto every environment except local. A candidate transport result without the same-record ztunnel denial is inconclusive; static cloud manifests also cannot replace a separately approved live EKS denial and positive-control procedure. Until that cloud procedure exists and produces fresh evidence, the decision remains NO LAUNCH.

Launch does not merge the provider, hosted-runtime, and generated-application planes. Every mapping is explicit, least privilege, audited, independently revocable, and proven with both positive and denied actions.

Decision rule: if any mandatory boundary lacks fresh evidence, do not launch. Withdraw traffic before runtime deletion, export and delete generated-app data through its owner, revoke grants independently, retain immutable audit, unenroll HelixWorks before shared mesh removal, then retire cluster and network in dependency order.

A production review previously stopped at infrastructure, workloads, and customer probes, leaving encrypted identity and waypoint enforcement as assumptions. The final gate now requires ambient behavior, ingress ownership, bounded proxy recovery, rollback, latency, and cost evidence from the exact EKS environment.

Production readiness is an evidence-backed, reversible customer decision.

• Launch only when all owners and customer probes agree.
• Pin all five release images; SNS/SQS owns cloud broker execution, while the inherited zero-replica broker image remains an immutable release input.
• Two contrasting applications prove shared platform invariants without pretending their schemas are identical.
• Bounded failure, recovery, rollback, export, and deletion are launch features.
• Ambient installation, enrollment, L4/L7 enforcement, proxy recovery, ingress ownership, and HelixWorks-first rollback are independent launch gates.

The course is complete when the learner can decide, operate, and recover without magical thinking.

• [ ] Dev, staging, and production use separate accounts, VPCs, states, roles, and EKS clusters
• [ ] Five staging-proven image digests reach production through reviewed overlays and Argo CD
• [ ] Three-plane positive and denied tests pass for two application archetypes
• [ ] Immutable revision, Terraform plan/state, AWS identity/context, five digests including broker, scans/signatures, Argo revision, and rollout identities are sealed
• [ ] Local bypass artifact is sealed, and a separate approved live EKS denial/control procedure exists and passes
• [ ] Admission, tenant denial, elastic-generation veto, restore RPO/RTO, explicit-token product traces, cost, and rollback evidence are sealed