Separate Infrastructure State from Workload State

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: infra/stacks/prod/main.tf

provider "aws" {
  region              = var.region
  allowed_account_ids = [var.account_id]
  assume_role {
    role_arn = var.deployer_role_arn
  }

Code to reality

Declared intent

Bind production infrastructure changes to one expected AWS account and an explicit deployer role.

Interpreter

Terraform configures the AWS provider, assumes the role, and rejects credentials for any other account.

Software effect

The plan is scoped to the production state and account boundary before resource reconciliation begins.

Hardware effect

Only an authorized apply can allocate billable AWS networking, control-plane, compute, and data resources.

Observable evidence

Caller identity, reviewed plan, remote state lock, and AWS resource tags agree on the production account.

The source tables below remain the detailed contract. Begin with these customer paths:

• D18-UC-01
  • Person: Platform engineer
  • Job: Assign one mutating owner to every infrastructure and workload resource
  • Observable result: Network, cluster, bootstrap, workload, and generated-app records have non-overlapping writers
• D18-UC-02
  • Person: Incident commander
  • Job: Find the failing boundary when a generated application is unavailable
  • Observable result: Diagnosis proceeds from AWS substrate to intended API, IaC state, Kubernetes, GitOps, and customer evidence

Now turn each customer job into a story with a result that an engineer can check:

• D18-US-01
  • Story: As a platform engineer, I want each resource assigned to one desired-state writer, so that control loops cannot continuously overwrite one another
  • Observable acceptance: Inventory names resource, environment, writer, state store, identity, deletion owner, and rejected conflict
• D18-US-02
  • Story: As an incident commander, I want boundary-specific observations in dependency order, so that I repair the real owner instead of trusting an…
  • Observable acceptance: Case contains freshness, expected and observed status, failing owner, bounded action, recovery, unaffected control, environment, time, run, and traces

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D18-FLOW-01
  • Trigger: Engineer proposes a new production workload
  • Responsible systems: IaC policy checker, Terraform or equivalent engine, cluster lifecycle, GitOps renderer/controller, generated-app owners
  • Authoritative state: Git plus IaC backend for infrastructure intent/mapping; Git plus Kubernetes API for workloads
  • Owned record: OwnershipAssignment
  • Observable evidence: Actor, resource inventory, writer, state store, environment, expected and observed policy, source revisions, timestamp, run, and evidence digest
  • Failure signal: Duplicate writer, missing deletion owner, shared production identity, or workload embedded in foundation state
• D18-FLOW-02
  • Trigger: Dev customer probe fails while GitOps last reported healthy
  • Responsible systems: AWS inventory, cluster API probe, IaC state reader, Kubernetes observers, GitOps controller, customer probe runner
  • Authoritative state: Provider APIs for resource reality; each controller’s store for its claim; Evidence Store for correlation
  • Owned record: InfrastructureIntent
  • Observable evidence: Boundary observations, stale or failed signal, owner-directed action, recovered response, positive control, environment, timestamps, run, cluster,…
  • Failure signal: Missing substrate, wrong API endpoint, stale state, unready workload, stale Git comparison, or failed response

Enterprise problem: Zheta Forge can report a synced workload while its cluster API or AWS substrate is unavailable, and two tools that both mutate one resource can fight forever or destroy the wrong layer. Whole-course context: The incoming evidence is a deterministic Kubernetes render with one Git owner; today places that workload inside the larger infrastructure chain. Today’s slice: We separate Infrastructure as Code (IaC) configuration and state, managed-cluster lifecycle, Kubernetes reconciliation, and GitOps workload ownership from local through production. End-of-day evidence: An ownership matrix and bounded dev failure show substrate existence, intended API reachability, IaC state agreement, Kubernetes convergence, Git revision, customer response, unaffected control, environment, time, run, and trace IDs independently. Still unsolved: Terraform’s concrete plan and state mechanics and AWS account implementation remain deferred.

“Desired state” is incomplete unless the team names which controller owns which resource and what its green signal proves. These use cases cover safe change and bottom-up diagnosis.

Declarative tools complement one another only when their responsibility boundaries are explicit. These stories make ownership conflict and scoped recovery observable.

A successful GitOps sync proves only a comparison against the Kubernetes API it reached; it cannot prove that customers, another API endpoint, or physical capacity are healthy. The flows record each boundary before reconciliation.

IaC, managed-cluster lifecycle, Kubernetes, and GitOps are not interchangeable automation choices; they own successive layers. A green signal is evidence only for the boundary that emitted it.

Calling Git the single source of truth hides provider reality and controller observations, so the platform must retain distinct records for intent, mapping, live state, and evidence. None silently overwrites another.

Generated-application database: Not created in this slice — durable infrastructure, ownership, controller, and evidence records are sufficient; existing application databases remain owned by their generated applications.

Local Kubernetes preserves control-loop behavior but collapses machine, disk, network, and power failure domains onto one laptop, so it is not production high-availability evidence. Dev, staging, and production use separate AWS accounts and clusters while promoting the same artifact and packaging model.

Diagnosis must observe before mutation because state surgery can erase the record needed for recovery. These commands compare independent owners in one named environment.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
terraform -chdir=infra/stacks/dev state list
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name zheta-forge-dev --alias zheta-forge-dev
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks describe-cluster --name zheta-forge-dev --query 'cluster.status'
kubectl --context zheta-forge-dev get nodes
kubectl --context zheta-forge-dev -n argocd get application forge-dev -o wide
test -n "${FORGE_URL:?set FORGE_URL to a reachable dev control-plane URL}"
FORGE_URL="$FORGE_URL" FORGE_TOKEN="${FORGE_TOKEN:?set the explicit bearer token}" ./scripts/smoke-product.sh

These exact paths and names come from infra/stacks/dev, infra/modules/environment, argocd/applicationsets/forge-dev.yaml, and scripts/smoke-product.sh. Because the EKS API is private-only, run aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig and every kubectl command from within the VPC or a connected network, as required by the Amazon EKS private-endpoint documentation.

One delivery system contains several desired-state owners, and none provides universal truth.

• Diagnose substrate, intended API, IaC mapping, Kubernetes, GitOps, then customer path.
• Give every resource one writer and one deletion owner.
• Local multi-node reconciliation is useful but does not prove independent failure domains.

The ownership model is safe when every repair has one direction.

• [ ] Infrastructure and workload resources have non-overlapping writers
• [ ] Environment inventory names account, cluster, network, identity, state, Git, secrets, backup, and deletion owners
• [ ] Read-only observations precede any repair or state surgery
• [ ] Recovery includes affected and unaffected customer evidence