Production EKS and the Protected Runtime Boundary

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: infra/modules/environment/main.tf

resource "aws_subnet" "private" {
  count             = 2
  vpc_id            = aws_vpc.this.id
  availability_zone = local.azs[count.index]
  cidr_block        = cidrsubnet(var.vpc_cidr, 4, count.index + 8)
  tags = merge(local.tags, {
    "kubernetes.io/role/internal-elb" = "1"
  })
}

Code to reality

Declared intent

Place environment workloads across two private availability-zone subnets.

Interpreter

Terraform and the AWS provider create subnets and label them for internal Kubernetes load balancers.

Software effect

EKS and its controllers discover private network placement from stable subnet IDs and tags.

Hardware effect

AWS allocates isolated address ranges and routes network interfaces across two physical zones.

Observable evidence

Terraform outputs, AWS subnet inventory, EKS node addresses, and a private connectivity probe agree.

The source tables below remain the detailed contract. Begin with these customer paths:

• D24-UC-01
  • Person: Production platform engineer
  • Job: Provision a protected multi-zone EKS runtime foundation
  • Observable result: Private cluster API, baseline capacity, encryption, audit logs, tags, budget, and Aurora deletion protection match approved policy
• D24-UC-02
  • Person: Production security approver
  • Job: Ensure only the production deployment role can mutate the cluster boundary
  • Observable result: Approved break-glass-free read succeeds and unapproved principals cannot administer the cluster

Now turn each customer job into a story with a result that an engineer can check:

• D24-US-01
  • Story: As a production platform engineer, I want policy-constrained EKS across independent zones, so that one capacity failure does not define the whole…
  • Observable acceptance: Plan and AWS inventory identify private endpoint, zones, capacity, KMS keys, logs, budgets, tags, and deletion safeguards
• D24-US-02
  • Story: As a production security approver, I want independently revocable production access, so that dev or staging authority cannot mutate customer runtime
  • Observable acceptance: Positive and negative checks name principal, action, resource, expected and observed outcome, account, time, and audit event ID

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D24-FLOW-01
  • Trigger: Production change board approves the foundation plan digest
  • Responsible systems: Terraform, VPC/endpoints, EKS, managed baseline node group, KMS, CloudWatch, budget, ECR, Aurora/Backup
  • Authoritative state: Production remote Terraform state and AWS resource APIs
  • Owned record: ProductionClusterRevision
  • Observable evidence: Approval, actor, account, plan digest, cluster ARN, node IDs/zones, KMS/log refs, environment, timestamp, and run ID
  • Failure signal: Wrong account, public endpoint, unhealthy node, encryption/logging gap, budget/tag failure, or Aurora deletion-policy violation
• D24-FLOW-02
  • Trigger: A staging principal attempts production cluster administration
  • Responsible systems: IAM, EKS access entries, Kubernetes RBAC/audit, CloudTrail
  • Authoritative state: IAM policies and AWS audit stores; Kubernetes API only after explicit mapping
  • Owned record: ProductionAccessGrant
  • Observable evidence: Denied principal/action/resource, positive control, account, cluster, expected and observed outcomes, time, and event ID
  • Failure signal: Unexpected access, shared role, stale session, missing audit event, or failed approved read

Enterprise problem: A production cluster can be technically identical to staging yet unsafe because deployment authority, failure domains, data protection, and deletion controls are weaker than the customer promise. Whole-course context: A staging dossier is an input only if it actually exists; today reviews the production foundation that may eventually receive a proven digest. Today’s slice: infra/stacks/prod derives zheta-forge-prod with a private EKS API, two private subnets, a three-node managed baseline, KMS Secrets encryption, five control-plane log types retained 365 days, a cost budget, immutable/scanned ECR, and deletion-protected Aurora with 35-day backups. EKS cluster deletion protection itself is not declared. End-of-day evidence: A real approved apply must prove these resources. Still unsolved: No AWS deployment or production launch evidence exists in this repository.

Production readiness starts below the application because every pod depends on network, compute, identity, and control-plane availability. No customer traffic is admitted in this slice.

Production controls must be observable before workloads arrive, or the first release becomes an unsafe infrastructure test. These stories make availability and authority falsifiable.

An approved Terraform apply creates infrastructure, not a launched product. The happy path deliberately stops at a protected empty cluster, and the denied path proves account isolation.

Managed EKS removes control-plane host operation, not customer accountability for worker, identity, networking, workload, and response health. Multi-zone capacity reduces one failure class but does not replace backup or disaster recovery.

Production infrastructure and access records need independent retention because deleting a cluster must not delete proof of who approved it. No generated application exists in production yet.

Generated-application database: Not created in this slice — the protected cluster foundation needs only infrastructure, access, approval, audit, and evidence state.

Applying workloads too early hides whether the substrate itself meets policy, so inspect the empty cluster first. Run this labelled snippet beside the public zheta-kubernetes-lab stable stack infra/stacks/prod; it deliberately inspects before applying gitops/apps/forge/overlays/prod.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name zheta-forge-prod --alias zheta-forge-prod
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks describe-cluster --name zheta-forge-prod --query 'cluster.{arn:arn,status:status,endpointPublicAccess:resourcesVpcConfig.endpointPublicAccess}'
kubectl --context zheta-forge-prod get nodes -L topology.kubernetes.io/zone
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" logs describe-log-groups --log-group-name-prefix /aws/eks/zheta-forge-prod/cluster
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" budgets describe-budgets --account-id "$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"

These checks align with the Terraform resources, not aspirations. AWS documents that selected EKS control-plane log types are delivered to CloudWatch and may take a few minutes to appear (EKS control-plane logging); private-only kubectl still requires VPC or connected-network access.

The empty cluster belongs only to the hosted-runtime boundary. The provider plane will later own release approval and collaborators; the generated-application plane will later own tenants and users; neither exists merely because a production IAM role or kubeconfig exists.

Decision rule: revoke access grants independently, retire traffic and generated-app state before workloads, drain workloads before cluster deletion, and remove the VPC only after cluster and managed-data dependencies disappear.

Production is a governed runtime boundary before it is a destination.

• An Active control plane does not prove usable nodes or customer health.
• Account, VPC, state, roles, and cluster are isolated from dev and staging.
• Availability zones, encryption, audit, budgets, and deletion safeguards are explicit contracts.

The foundation is ready for bootstrap only after policy and denial proofs agree.

• [ ] Production account, state, VPC, role, and cluster are unique
• [ ] API is private and baseline nodes span approved zones
• [ ] Encryption, audit logs, tags, budgets, and protection are visible
• [ ] Staging principal is denied while production verifier succeeds