Observe, Back Up, Restore, and Reconstruct

Enterprise problem: Metrics can show a healthy pod while customer data is unrecoverable, and a backup can exist while restore credentials, keys, or procedures fail. Whole-course context: The incoming runtime scales workloads and machines with bounded evidence; today protects its durable state and operational knowledge. Today’s slice: We correlate metrics, logs, and traces; back up Aurora and object artifacts; rehearse restore into isolated staging; and reconstruct VPC, EKS, Argo CD, workloads, and customer flow from authoritative sources. End-of-day evidence: Recovery point, restore resource IDs, Git/Terraform revisions, cluster and workload identities, positive/negative customer probes, recovery times, environment, timestamp, run, and trace IDs form a disaster-recovery dossier. Still unsolved: Production launch approval and live traffic remain deferred.

The customer outcome is a reviewable observe, back up, restore, and reconstruct change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.

Metrics/log/trace stack, backup controller, restore controller, and incident runbook; pods/nodes/object storage; application databases, etcd snapshots, backup bucket, and evidence store; cache: rebuild or invalidate after restore.

This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.

Run read-only evidence checks first

Mutation before observation can destroy the clues needed to select the right owner. Run this labelled snippet from the public monorepo using stable paths infra/stacks/prod, argocd/applicationsets/forge-prod.yaml, and scripts/smoke-product.sh.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
terraform -chdir=infra/stacks/prod state list
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name helixworks-forge-prod --alias helixworks-forge-prod
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks describe-cluster --name helixworks-forge-prod --query 'cluster.status'
kubectl --context helixworks-forge-prod get nodes
kubectl --context helixworks-forge-prod -n argocd get application forge-prod -o wide
test -n "${FORGE_URL:?set FORGE_URL to the explicit reachable production control-plane URL}"
FORGE_URL="$FORGE_URL" FORGE_TOKEN="${FORGE_TOKEN:?set the explicit bearer token}" ./scripts/smoke-product.sh

The current Terraform module actually declares three Aurora clusters, versioned encrypted S3, a KMS-encrypted AWS Backup vault, daily backup plan, and production 35-day retention. It does not prove a restore; launch remains blocked until a separate-account restore run records integrity, authorization, RPO, and RTO.

Support access to provider evidence does not grant runtime secret access or generated-app row access. Restore roles are temporary, account-scoped, audited, and independently revocable.

Decision rule: restore source-owned enterprise data only through its owner; validate generated-app authorization before use; delete rehearsal data and runtime after proof while retaining the immutable recovery dossier.

Recovery is a customer workflow reconstructed across multiple owners.

• Observe bottom-up before mutation or state surgery.
• Test restore, integrity, authorization, RPO, and RTO in isolation.
• Metrics, logs, traces, backups, Terraform, Git, and controllers complement rather than replace one another.

The platform is recoverable only when the rehearsal ends in customer evidence.

• [ ] Positive and denied support queries preserve tenant scope
• [ ] Recovery point, key, schema, and checksum are verified
• [ ] Reconstruction follows substrate to API to state to workload to customer
• [ ] Rehearsal copy is erased and dossier retained