Observe, Back Up, Restore, and Reconstruct

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: services/evidence/app.py

for receipt, event in BROKER.receive("evidence"):
                event["evidence_id"] = event.get("evidence_id") or __import__("hashlib").sha256(__import__("json").dumps(event, sort_keys=True).encode()).hexdigest()
                STORE.append(event)
                BROKER.acknowledge("evidence", receipt)

Code to reality

Declared intent

Receive broker deliveries, append idempotent evidence, acknowledge only after success, and expose tenant-scoped queries.

Interpreter

A background subscriber reads local HTTP-broker or SQS receipts; the SQL store deduplicates by evidence ID before acknowledgement.

Software effect

Append failure leaves a message available for redelivery; repeated delivery yields one durable event returned by an authenticated query.

Hardware effect

The evidence process consumes CPU for hashing and commits database pages to durable storage.

Observable evidence

Receipt and acknowledgement observations, forced redelivery, one stored evidence ID, authenticated organization query, and restart prove the contract.

The source tables below remain the detailed contract. Begin with these customer paths:

• D29-UC-01
  • Person: Support engineer
  • Job: Diagnose a failed generated-app action across services
  • Observable result: One trace joins customer request, logs, metrics, release, actor, and owned records without exposing secrets
• D29-UC-02
  • Person: Disaster-recovery commander
  • Job: Restore durable data and reconstruct the platform in an isolated account
  • Observable result: Restored tenant and artifact pass integrity and lifecycle probes within RPO/RTO

Now turn each customer job into a story with a result that an engineer can check:

• D29-US-01
  • Story: As a support engineer, I want tenant-scoped traces connected to releases and services, so that I can diagnose without broad data access
  • Observable acceptance: Accepted/denied queries record actor, tenant, time window, trace, release, service, result, environment, and audit ID
• D29-US-02
  • Story: As a disaster-recovery commander, I want a rehearsed bottom-up reconstruction, so that backup claims become customer-visible recovery
  • Observable acceptance: Dossier contains recovery point, Terraform/Git revisions, restored IDs, integrity checks, RPO/RTO, affected replay, unaffected control, and run

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D29-FLOW-01
  • Trigger: Support receives a failed publish evidence ID
  • Responsible systems: OpenTelemetry instrumentation/collector, metrics/log/trace stores, Forge services, release/evidence stores
  • Authoritative state: Each service store for domain state; telemetry stores for observations; provider audit for access
  • Owned record: RecoveryPoint
  • Observable evidence: Actor, tenant, trace, release, observations, owner-directed action, recovered response, environment, time, and audit ID
  • Failure signal: Broken correlation, stale signal, secret/tenant leakage, wrong release, or failed replay
• D29-FLOW-02
  • Trigger: Scheduled isolated recovery rehearsal begins
  • Responsible systems: Backup service, Aurora/S3/KMS, Terraform, EKS, Argo bootstrap, GitOps, Kubernetes, probe runner
  • Authoritative state: Backup catalog/data stores, Terraform state/Git, AWS/Kubernetes APIs, application/evidence stores
  • Owned record: RestoreRun
  • Observable evidence: Recovery point, restored resources, revisions, identities, integrity, probes, RPO/RTO, production control, environment, time, and run ID
  • Failure signal: Missing key/snapshot, integrity mismatch, stale state, unavailable API, unready workload, wrong revision, or failed customer probe

Enterprise problem: Metrics can show a healthy pod while customer data is unrecoverable, and a backup can exist while restore credentials, keys, or procedures fail. Whole-course context: The incoming runtime scales workloads and machines with bounded evidence; today protects its durable state and operational knowledge. Today’s slice: We correlate metrics, logs, and traces; back up Aurora and object artifacts; rehearse restore into isolated staging; and reconstruct VPC, EKS, Argo CD, workloads, and customer flow from authoritative sources. End-of-day evidence: Recovery point, restore resource IDs, Git/Terraform revisions, cluster and workload identities, positive/negative customer probes, recovery times, environment, timestamp, run, and trace IDs form a disaster-recovery dossier. Still unsolved: Production launch approval and live traffic remain deferred.

A backup is useful only if an accountable operator can restore a consistent customer workflow within objectives. Recovery point objective (RPO) bounds tolerable data loss; recovery time objective (RTO) bounds restoration time.

Observability and recoverability are related but not interchangeable: telemetry explains behavior, while backup and declarative sources recreate state. These stories require both.

Recovery must start from a declared customer impact and proceed through substrate, API, state, workloads, then product evidence. Restoring only one database is not a complete application recovery.

Different desired-state owners require a bottom-up order: AWS substrate, intended EKS API, Terraform mapping, Kubernetes controllers, GitOps revision, then customer response. A green upper layer cannot override a failed lower layer.

Backups cannot merge provider metadata, runtime state, generated-app rows, and source-enterprise data into one unscoped archive. Each owner defines retention, encryption, restore, export, and deletion.

Generated-application database: Required in this slice — isolated restore proves generated-app tenant data, constraints, authorization predicates, and deletion independently of provider records.

Mutation before observation can destroy the clues needed to select the right owner. Run this labelled snippet from the public monorepo using stable paths infra/stacks/prod, argocd/applicationsets/forge-prod.yaml, and scripts/smoke-product.sh.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
terraform -chdir=infra/stacks/prod state list
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name zheta-forge-prod --alias zheta-forge-prod
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks describe-cluster --name zheta-forge-prod --query 'cluster.status'
kubectl --context zheta-forge-prod get nodes
kubectl --context zheta-forge-prod -n argocd get application forge-prod -o wide
test -n "${FORGE_URL:?set FORGE_URL to the explicit reachable production control-plane URL}"
FORGE_URL="$FORGE_URL" FORGE_TOKEN="${FORGE_TOKEN:?set the explicit bearer token}" ./scripts/smoke-product.sh

The current Terraform module actually declares three Aurora clusters, versioned encrypted S3, a KMS-encrypted AWS Backup vault, daily backup plan, and production 35-day retention. It does not prove a restore; launch remains blocked until a separate-account restore run records integrity, authorization, RPO, and RTO.

Support access to provider evidence does not grant runtime secret access or generated-app row access. Restore roles are temporary, account-scoped, audited, and independently revocable.

Decision rule: restore source-owned enterprise data only through its owner; validate generated-app authorization before use; delete rehearsal data and runtime after proof while retaining the immutable recovery dossier.

Recovery is a customer workflow reconstructed across multiple owners.

• Observe bottom-up before mutation or state surgery.
• Test restore, integrity, authorization, RPO, and RTO in isolation.
• Metrics, logs, traces, backups, Terraform, Git, and controllers complement rather than replace one another.

The platform is recoverable only when the rehearsal ends in customer evidence.

• [ ] Positive and denied support queries preserve tenant scope
• [ ] Recovery point, key, schema, and checksum are verified
• [ ] Reconstruction follows substrate to API to state to workload to customer
• [ ] Rehearsal copy is erased and dossier retained