Separate Infrastructure State from Workload State

Enterprise problem: HelixWorks can report a synced workload while its cluster API or AWS substrate is unavailable, and two tools that both mutate one resource can fight forever or destroy the wrong layer. Whole-course context: The incoming evidence is a deterministic Kubernetes render with one Git owner; today places that workload inside the larger infrastructure chain. Today’s slice: We separate Infrastructure as Code (IaC) configuration and state, managed-cluster lifecycle, Kubernetes reconciliation, and GitOps workload ownership from local through production. End-of-day evidence: An ownership matrix and bounded dev failure show substrate existence, intended API reachability, IaC state agreement, Kubernetes convergence, Git revision, customer response, unaffected control, environment, time, run, and trace IDs independently. Still unsolved: Terraform’s concrete plan and state mechanics and AWS account implementation remain deferred.

The customer outcome is a reviewable separate infrastructure state from workload state change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.

Terraform/IaC engine and provider API, Kubernetes API/controllers, and Argo CD; cloud VMs/network plus cluster workloads; Terraform state backend, Git, and Kubernetes etcd; cache: not involved.

This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.

Compare the same layers across environments

Local Kubernetes preserves control-loop behavior but collapses machine, disk, network, and power failure domains onto one laptop, so it is not production high-availability evidence. Dev, staging, and production use separate AWS accounts and clusters while promoting the same artifact and packaging model.

Diagnosis must observe before mutation because state surgery can erase the record needed for recovery. These commands compare independent owners in one named environment.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
terraform -chdir=infra/stacks/dev state list
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name helixworks-forge-dev --alias helixworks-forge-dev
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks describe-cluster --name helixworks-forge-dev --query 'cluster.status'
kubectl --context helixworks-forge-dev get nodes
kubectl --context helixworks-forge-dev -n argocd get application forge-dev -o wide
test -n "${FORGE_URL:?set FORGE_URL to a reachable dev control-plane URL}"
FORGE_URL="$FORGE_URL" FORGE_TOKEN="${FORGE_TOKEN:?set the explicit bearer token}" ./scripts/smoke-product.sh

These exact paths and names come from infra/stacks/dev, infra/modules/environment, argocd/applicationsets/forge-dev.yaml, and scripts/smoke-product.sh. Because the EKS API is private-only, run aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig and every kubectl command from within the VPC or a connected network, as required by the Amazon EKS private-endpoint documentation.

One delivery system contains several desired-state owners, and none provides universal truth.

• Diagnose substrate, intended API, IaC mapping, Kubernetes, GitOps, then customer path.
• Give every resource one writer and one deletion owner.
• Local multi-node reconciliation is useful but does not prove independent failure domains.

The ownership model is safe when every repair has one direction.

• [ ] Infrastructure and workload resources have non-overlapping writers
• [ ] Environment inventory names account, cluster, network, identity, state, Git, secrets, backup, and deletion owners
• [ ] Read-only observations precede any repair or state surgery
• [ ] Recovery includes affected and unaffected customer evidence