Observe, Migrate, Recover, and Map Ambient Mesh to EKS

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: scripts/failure-istio-ambient.sh

: "${TARGET_POD:?set TARGET_POD to the reviewed pod name}"
: "${TARGET_POD_UID:?set TARGET_POD_UID to the reviewed pod UID}"

positive_control() {
  kubectl --context "$MESH_CONTEXT" get --raw=/readyz >/dev/null
  [ "$(kubectl --context "$MESH_CONTEXT" -n zheta-forge get deployment/control-plane -o jsonpath='{.status.availableReplicas}')" -ge 1 ]
}

Code to reality

Declared intent

Require one reviewed Pod name and immutable UID while preserving API and product availability before bounded failure.

Interpreter

The shell validates context, approval, UID, label, owner, and cardinality before kubectl invokes the owning controller path.

Software effect

Only the exact approved proxy can terminate; one different replacement UID must appear and product policy probes must still pass.

Hardware effect

The selected node stops and starts a proxy process, reloads image layers, and re-establishes encrypted connections.

Observable evidence

Name/UID, label, owner, unique replacement, API readiness, control-plane availability, allowed POST, denied GET, and recovery time prove recovery.

The source tables below remain the detailed contract. Begin with these customer paths:

• D34-UC-01
  • Person: Enterprise builder
  • Job: Continue generating and previewing applications while mesh infrastructure changes
  • Observable result: Approved Forge request and intentional denial remain correct before and after one proxy replacement
• D34-UC-02
  • Person: Platform migration commander
  • Job: Unenroll Forge or remove shared mesh components with bounded ownership
  • Observable result: Forge labels/policies are removed first; shared releases remain unless separately approved

Now turn each customer job into a story with a result that an engineer can check:

• D34-US-01
  • Story: As an enterprise builder, I want proxy failure recovery proven with my product request, so that controller health is connected to customer outcome
  • Observable acceptance: Evidence correlates old/new proxy UIDs, recovery time, allowed and denied traces, source revision, environment, and run
• D34-US-02
  • Story: As a migration commander, I want application unenrollment separated from shared control-plane removal, so that rollback does not delete another…
  • Observable acceptance: Context and approval gates, removed Forge labels/policies, preserved releases/CRDs, and post-rollback request are recorded

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D34-FLOW-01
  • Trigger: Approved operator selects ztunnel-recovery or waypoint-recovery
  • Responsible systems: ztunnel DaemonSet, waypoint Gateway controller, Kubernetes scheduler/kubelet, Istiod, Forge services
  • Authoritative state: Kubernetes API for proxy desired/observed state; Forge/evidence stores for customer result
  • Owned record: MeshMigrationRevision
  • Observable evidence: Exact target, old/new UID, DaemonSet or Gateway status, recovery time, product traces, environment, time, incident ID
  • Failure signal: Empty/ambiguous target, unreplaced Pod, unprogrammed Gateway, changed denial semantics, or failed replay
• D34-FLOW-02
  • Trigger: Migration commander invokes bounded Forge rollback
  • Responsible systems: namespace labels, Forge Gateway/policies, Helm release owners, Gateway API owner
  • Authoritative state: Git and Kubernetes API for Forge state; Helm release metadata for shared mesh; external approval record
  • Owned record: ProxyRecoveryEvent
  • Observable evidence: Approval, context, removed objects, preserved shared resources, customer response, owner, environment, time, rollback ID
  • Failure signal: Wrong context, missing approval, shared release removal without separate flag, CRD deletion, or broken direct path

Enterprise problem: A mesh can report healthy controllers while customer traffic is denied, bypasses a waypoint, or depends on one failed proxy; uninstalling shared infrastructure during rollback can widen the outage. Whole-course context: The incoming Forge path has ambient enrollment, strict mTLS, ServiceAccount L4 policy, and destination L7 policy, but no production claim. Today’s slice: We observe each boundary, inject one approved ztunnel or waypoint failure, verify controller recovery, and unenroll Forge separately from shared Istio removal. We map the same responsibilities to private Amazon EKS without pretending local Kind evidence proves AWS. End-of-day evidence: Context, versions, proxy/workload status, policy, positive/denied traces, old/new proxy UIDs, recovery time, rollback result, unaffected control, environment, run, and cost owner form the migration dossier. Still unsolved: Day 35 keeps production blocked until real EKS and customer evidence agree.

Controller readiness is not customer readiness, so migration must preserve the full Forge lifecycle through normal, failed, recovered, and rolled-back states.

Recovery without a product replay can restore a proxy while leaving the customer path broken.

A useful failure lab resolves one exact target and preserves an unaffected control, avoiding broad chaos that teaches little.

Different controllers repair different failures, so diagnosis climbs from nodes and proxies through policy to the product response.

Migration decisions need durable correlation, but they must not take ownership of application data or shared cluster resources.

Generated-application database: Not created in this slice — generated applications keep their tenant data while runtime and evidence owners record mesh migration state.

Deleting an arbitrary proxy can affect an unknown blast radius, so the checked-in script requires the exact context, explicit approval, and one nonempty labelled target.

Run recovery only after the staged evidence is sealed. Day 32 must contain the paired admission NetworkPolicies, reviewed evidence/generator/broker Pod UIDs, independent kindnet timeout, an HTTP GET/send/read that returns no response bytes, a candidate timeout/reset/broken-pipe/abort/zero-byte EOF, one destination-ztunnel record containing target IP:port plus evidence identity plus denial marker, and control-plane POST before/after. Day 33 must contain the Accepted/Programmed waypoint, waypoint-only L4 rewrite, executable POST success, and executable GET 403. Rendered policy, a candidate transport failure, and proxy status are not substitutes for the same-record denial plus live controls.

: "${MESH_CONTEXT:?set the exact kubectl context}"
kubectl --context "$MESH_CONTEXT" -n istio-system get pods -l app=ztunnel -o custom-columns=NAME:.metadata.name,UID:.metadata.uid,NODE:.spec.nodeName
MESH_FAILURE_APPROVED=1 TARGET_POD="${TARGET_POD:?review one exact name}" TARGET_POD_UID="${TARGET_POD_UID:?record its immutable UID}" ./scripts/failure-istio-ambient.sh ztunnel-recovery
# Or test the destination proxy independently:
MESH_FAILURE_APPROVED=1 TARGET_POD="$TARGET_POD" TARGET_POD_UID="$TARGET_POD_UID" ./scripts/failure-istio-ambient.sh waypoint-recovery
./scripts/verify-istio-ambient.sh forge
# Replay an approved Forge request and one expected denial before closing evidence.

Removing shared Istio first can strand enrolled traffic and affect other namespaces, so rollback defaults to Forge-only unenrollment.

: "${MESH_CONTEXT:?set the exact kubectl context}"
MESH_ROLLBACK_APPROVED=1 ./scripts/rollback-istio-ambient.sh

The checked-in rollback supports only the local overlay and vetoes any other environment. It removes Forge enrollment and Forge-owned mesh resources, restores the local base NetworkPolicies, waits for all product Deployments, and proves an internal generation request. Shared Istio releases and Gateway API CRDs always remain because another owner may use them. It does not prove an AWS rollback; EKS requires a separately implemented and approved procedure against the intended private cluster, node groups, identities, and billing boundary.

Local Kind nodes collapse power, disk, and network failure domains onto one laptop, so their proxy recovery demonstrates reconciliation but not production availability.

A static mesh configuration previously had no bounded failure or exit proof, so operators could not distinguish recoverable proxy loss from a customer outage. The migrated state has an approved recovery loop and a Forge-first rollback boundary.

Mesh migration is complete only when normal traffic, denial, failure recovery, and rollback all have product evidence.

• Diagnose substrate and proxies before policy and product symptoms.
• Delete one validated target, wait for its actual owner, then replay customer behavior.
• Unenroll the application before considering removal of shared cluster capability.
• Map local control relationships to EKS without promoting local evidence into an AWS claim.

The migration dossier must be complete enough for Day 35 to veto production when any boundary is stale.

• [ ] Exact context, source, versions, approval, and healthy precondition recorded
• [ ] Local bypass run seals admitted transport, reviewed Pod UIDs, direct denial, destination-ztunnel observation, and before/after waypoint controls
• [ ] One ztunnel or waypoint failure recovered with old/new UIDs
• [ ] Approved and denied Forge traces replayed with unaffected control
• [ ] Forge-only rollback preserves shared releases and Gateway API CRDs
• [ ] EKS owners, private access, worker capacity, observability, and cost gaps named