Production EKS and the Protected Runtime Boundary

Enterprise problem: A production cluster can be technically identical to staging yet unsafe because deployment authority, failure domains, data protection, and deletion controls are weaker than the customer promise. Whole-course context: A staging dossier is an input only if it actually exists; today reviews the production foundation that may eventually receive a proven digest. Today’s slice: infra/stacks/prod derives helixworks-forge-prod with a private EKS API, two private subnets, a three-node managed baseline, KMS Secrets encryption, five control-plane log types retained 365 days, a cost budget, immutable/scanned ECR, and deletion-protected Aurora with 35-day backups. EKS cluster deletion protection itself is not declared. End-of-day evidence: A real approved apply must prove these resources. Still unsolved: No AWS deployment or production launch evidence exists in this repository.

The customer outcome is a reviewable production eks and the protected runtime boundary change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.

Production EKS, Argo CD, admission policy, ingress, and observability stack; multi-AZ nodes/pods/load balancers; production state backend, etcd/backup stores, registry, and audit storage; cache: only application-owned and independently operated.

This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.

Inspect before admitting workloads

Applying workloads too early hides whether the substrate itself meets policy, so inspect the empty cluster first. Run this labelled snippet beside the public helixworks-kubernetes-lab stable stack infra/stacks/prod; it deliberately inspects before applying gitops/apps/forge/overlays/prod.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name helixworks-forge-prod --alias helixworks-forge-prod
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks describe-cluster --name helixworks-forge-prod --query 'cluster.{arn:arn,status:status,endpointPublicAccess:resourcesVpcConfig.endpointPublicAccess}'
kubectl --context helixworks-forge-prod get nodes -L topology.kubernetes.io/zone
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" logs describe-log-groups --log-group-name-prefix /aws/eks/helixworks-forge-prod/cluster
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" budgets describe-budgets --account-id "$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"

These checks align with the Terraform resources, not aspirations. AWS documents that selected EKS control-plane log types are delivered to CloudWatch and may take a few minutes to appear (EKS control-plane logging); private-only kubectl still requires VPC or connected-network access.

The empty cluster belongs only to the hosted-runtime boundary. The provider plane will later own release approval and collaborators; the generated-application plane will later own tenants and users; neither exists merely because a production IAM role or kubeconfig exists.

Decision rule: revoke access grants independently, retire traffic and generated-app state before workloads, drain workloads before cluster deletion, and remove the VPC only after cluster and managed-data dependencies disappear.

Production is a governed runtime boundary before it is a destination.

• An Active control plane does not prove usable nodes or customer health.
• Account, VPC, state, roles, and cluster are isolated from dev and staging.
• Availability zones, encryption, audit, budgets, and deletion safeguards are explicit contracts.

The foundation is ready for bootstrap only after policy and denial proofs agree.

• [ ] Production account, state, VPC, role, and cluster are unique
• [ ] API is private and baseline nodes span approved zones
• [ ] Encryption, audit logs, tags, budgets, and protection are visible
• [ ] Staging principal is denied while production verifier succeeds