Isolate AWS Accounts, IAM, State, and Guardrails

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: infra/stacks/prod/main.tf

provider "aws" {
  region              = var.region
  allowed_account_ids = [var.account_id]
  assume_role {
    role_arn = var.deployer_role_arn
  }

Code to reality

Declared intent

Bind production infrastructure changes to one expected AWS account and an explicit deployer role.

Interpreter

Terraform configures the AWS provider, assumes the role, and rejects credentials for any other account.

Software effect

The plan is scoped to the production state and account boundary before resource reconciliation begins.

Hardware effect

Only an authorized apply can allocate billable AWS networking, control-plane, compute, and data resources.

Observable evidence

Caller identity, reviewed plan, remote state lock, and AWS resource tags agree on the production account.

The source tables below remain the detailed contract. Begin with these customer paths:

• D20-UC-01
  • Person: Release manager
  • Job: Promote one tested artifact digest from dev to staging to production
  • Observable result: Same Research Brief, Service Desk, or Field Inspection bytes cross explicit evidence gates into separate accounts and clusters
• D20-UC-02
  • Person: Cloud security administrator
  • Job: Prevent a dev identity or state runner from reading or changing production
  • Observable result: Dev role cannot assume production execution, access production state, secrets, cluster API, or deletion operations

Now turn each customer job into a story with a result that an engineer can check:

• D20-US-01
  • Story: As a release manager, I want to promote the same immutable digest through isolated accounts, so that environment confidence grows without rebuilding…
  • Observable acceptance: Each gate records digest, source, tests, approver, account, cluster, expected and observed response, timestamp, run, and trace
• D20-US-02
  • Story: As a cloud security administrator, I want environment roles and state access independently scoped, so that a dev compromise cannot become a…
  • Observable acceptance: Denied role assumption, state read, secret read, and cluster access are recorded while approved production runner and customer probe pass

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D20-FLOW-01
  • Trigger: Release manager approves a dev-tested artifact for staging
  • Responsible systems: Artifact Registry, evidence gate, environment-specific release roles, per-account EKS and runtime controllers, probe runner
  • Authoritative state: Release Store for promotion; digest registry for bytes; each cluster API for runtime
  • Owned record: EnvironmentInventory
  • Observable evidence: Actor, source, digest, registry scan, test evidence, account and cluster IDs, expected and observed results, approvals, environments, timestamps,…
  • Failure signal: Digest changed, evidence incomplete, wrong account role, cluster mismatch, or customer probe failed
• D20-FLOW-02
  • Trigger: Dev runner attempts production role, state, secret, and cluster access
  • Responsible systems: IAM Identity Center or approved federation, Security Token Service, environment role policies, state bucket and key policies, secrets and EKS…
  • Authoritative state: AWS IAM and resource policies for authority; audit service for attempts
  • Owned record: EnvironmentRole
  • Observable evidence: Caller ARN, denied actions and resource ARNs, account IDs, audit event IDs, production read and response traces, environment, time, and run ID
  • Failure signal: Dev principal assumes production role, reads production backend, reaches cluster, reads secret, or audit event missing

Enterprise problem: If dev, staging, and production share an AWS account, Elastic Kubernetes Service (EKS) cluster, Terraform state, or broad administrator role, one mistaken variable or compromised runner can cross every environment and erase the evidence needed for recovery. Whole-course context: The incoming evidence is a Terraform change model with modules, locked state, physical effects, and guarded repair; today implements the maximum-isolation topology chosen for Zheta Forge. Today’s slice: We assign dedicated AWS accounts, Virtual Private Clouds (VPCs), EKS clusters, state backends, short-lived Identity and Access Management (IAM) roles, secrets, observability, backups, deletion authority, and promotion gates to dev, staging, and production while local remains non-cloud. End-of-day evidence: An environment matrix, role-denial test, backend separation proof, cluster identities, artifact-digest promotion chain, customer probes, unaffected control, timestamps, run, and trace IDs demonstrate isolation. Still unsolved: The next slice will bootstrap and reconcile workloads through Argo CD, a GitOps controller, inside each isolated environment.

Environment names are labels unless credentials, failure domains, state, and deletion authority are actually separated. These use cases prove both intended promotion and blocked cross-environment access.

Separate accounts reduce blast radius only when trust policies and state access do not recreate a hidden bridge. These stories bind promotion and denial to observable account identities.

Copying source into production and rebuilding there can change dependencies or policy inputs, so promotion must move immutable identity plus evidence. The denial flow tests real AWS authorization rather than trusting an architecture diagram.

Maximum isolation requires more than separate namespaces because namespaces share account IAM, cluster control plane, network, state, and failure domains. Zheta Forge uses one AWS account and one EKS cluster for each cloud environment.

An environment matrix kept only in documentation drifts from deployed accounts and roles. Durable inventory and evidence records constrain automation while AWS APIs remain authoritative for live resources.

Generated-application database: Not created in this slice — the slice creates isolated infrastructure, identity, state, and promotion records; generated-app databases are later deployed inside each runtime boundary and retain their own tenant authority.

Isolation becomes actionable when each paid or sensitive resource has one accountable owner and deletion order. Local uses Kind and local secrets only; every cloud environment has its own account and cluster.

Policy review cannot prove the deployed trust relationship, so use a bounded dev identity to request explicitly forbidden production access. These commands must run in an approved test harness and are read-only or expected to fail.

: "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_DEV_ACCOUNT_ID:?}" "${EXPECTED_PROD_ACCOUNT_ID:?}"
dev_account="$(aws --profile "${DEV_PROFILE:?set the approved dev profile}" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$dev_account" = "$EXPECTED_DEV_ACCOUNT_ID" ] || { echo "dev account mismatch" >&2; exit 1; }
aws --profile "$DEV_PROFILE" --region "$AWS_REGION" sts assume-role --role-arn "${PROD_TERRAFORM_ROLE_ARN:?}" --role-session-name denied-test
aws --profile "$DEV_PROFILE" --region "$AWS_REGION" s3api head-object --bucket "${PROD_STATE_BUCKET:?}" --key "${PROD_STATE_KEY:?use the approved backend.hcl key}"
prod_account="$(aws --profile "${PROD_READ_PROFILE:?}" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$prod_account" = "$EXPECTED_PROD_ACCOUNT_ID" ] || { echo "prod account mismatch" >&2; exit 1; }
aws --profile "$PROD_READ_PROFILE" --region "$AWS_REGION" eks describe-cluster --name zheta-forge-prod --query 'cluster.arn'
test -n "${FORGE_URL:?set FORGE_URL to a reachable production control-plane URL}"
FORGE_URL="$FORGE_URL" FORGE_TOKEN="${FORGE_TOKEN:?set the explicit bearer token}" ./scripts/smoke-product.sh

The exact environment root is infra/stacks/prod, and the module derives cluster name zheta-forge-prod; backend profiles and keys remain operator-supplied through backend.hcl, so the snippet requires them explicitly. The smoke script defaults to localhost only for local Compose; cloud proof must set an explicit reachable FORGE_URL.

The two dev-to-production calls must return access denied; the approved production read-only identity and customer probe are positive controls. Never grant temporary production access merely to make the test convenient.

Maximum isolation uses separate accounts, clusters, state, identities, and approvals while reusing modules and immutable artifacts.

• Namespaces do not isolate AWS IAM, Terraform state, cluster control plane, or account blast radius.
• Promotion moves a digest and evidence; it never rebuilds application bytes.
• A real access-denied test plus positive control proves deployed guardrails.

An AWS environment is isolated only when both intended access and denied cross-access are observable.

• [ ] Dev, staging, and production have distinct accounts, VPCs, EKS clusters, state keys, and roles
• [ ] No environment runner holds long-lived cloud or kubeconfig credentials
• [ ] Promotion preserves one signed artifact digest through independent gates
• [ ] Cross-account denials and approved production positive controls are recorded