Observe, Migrate, Recover, and Map Ambient Mesh to EKS

Enterprise problem: A mesh can report healthy controllers while customer traffic is denied, bypasses a waypoint, or depends on one failed proxy; uninstalling shared infrastructure during rollback can widen the outage. Whole-course context: The incoming HelixWorks path has ambient enrollment, strict mTLS, ServiceAccount L4 policy, and destination L7 policy, but no production claim. Today’s slice: We observe each boundary, inject one approved ztunnel or waypoint failure, verify controller recovery, and unenroll HelixWorks separately from shared Istio removal. We map the same responsibilities to private Amazon EKS without pretending local Kind evidence proves AWS. End-of-day evidence: Context, versions, proxy/workload status, policy, positive/denied traces, old/new proxy UIDs, recovery time, rollback result, unaffected control, environment, run, and cost owner form the migration dossier. Still unsolved: Day 35 keeps production blocked until real EKS and customer evidence agree.

The customer outcome is a reviewable observe, migrate, recover, and map ambient mesh to eks change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.

Argo CD, Istio ambient components, EKS API, rollback controller/runbook, and observability stack; nodes/pods/load balancers; Git, Kubernetes etcd, registry, backups, and evidence store; cache: invalidate derived entries after rollback.

This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.

Run the bounded recovery loop

Deleting an arbitrary proxy can affect an unknown blast radius, so the checked-in script requires the exact context, explicit approval, and one nonempty labelled target.

Run recovery only after the staged evidence is sealed. Day 32 must contain the paired admission NetworkPolicies, reviewed evidence/generator/broker Pod UIDs, independent kindnet timeout, an HTTP GET/send/read that returns no response bytes, a candidate timeout/reset/broken-pipe/abort/zero-byte EOF, one destination-ztunnel record containing target IP:port plus evidence identity plus denial marker, and control-plane POST before/after. Day 33 must contain the Accepted/Programmed waypoint, waypoint-only L4 rewrite, executable POST success, and executable GET 403. Rendered policy, a candidate transport failure, and proxy status are not substitutes for the same-record denial plus live controls.

: "${MESH_CONTEXT:?set the exact kubectl context}"
kubectl --context "$MESH_CONTEXT" -n istio-system get pods -l app=ztunnel -o custom-columns=NAME:.metadata.name,UID:.metadata.uid,NODE:.spec.nodeName
MESH_FAILURE_APPROVED=1 TARGET_POD="${TARGET_POD:?review one exact name}" TARGET_POD_UID="${TARGET_POD_UID:?record its immutable UID}" ./scripts/failure-istio-ambient.sh ztunnel-recovery
# Or test the destination proxy independently:
MESH_FAILURE_APPROVED=1 TARGET_POD="$TARGET_POD" TARGET_POD_UID="$TARGET_POD_UID" ./scripts/failure-istio-ambient.sh waypoint-recovery
./scripts/verify-istio-ambient.sh forge
# Replay an approved HelixWorks request and one expected denial before closing evidence.

Removing shared Istio first can strand enrolled traffic and affect other namespaces, so rollback defaults to HelixWorks-only unenrollment.

: "${MESH_CONTEXT:?set the exact kubectl context}"
MESH_ROLLBACK_APPROVED=1 ./scripts/rollback-istio-ambient.sh

The checked-in rollback supports only the local overlay and vetoes any other environment. It removes HelixWorks enrollment and HelixWorks-owned mesh resources, restores the local base NetworkPolicies, waits for all product Deployments, and proves an internal generation request. Shared Istio releases and Gateway API CRDs always remain because another owner may use them. It does not prove an AWS rollback; EKS requires a separately implemented and approved procedure against the intended private cluster, node groups, identities, and billing boundary.

Local Kind nodes collapse power, disk, and network failure domains onto one laptop, so their proxy recovery demonstrates reconciliation but not production availability.

A static mesh configuration previously had no bounded failure or exit proof, so operators could not distinguish recoverable proxy loss from a customer outage. The migrated state has an approved recovery loop and a HelixWorks-first rollback boundary.

Mesh migration is complete only when normal traffic, denial, failure recovery, and rollback all have product evidence.

• Diagnose substrate and proxies before policy and product symptoms.
• Delete one validated target, wait for its actual owner, then replay customer behavior.
• Unenroll the application before considering removal of shared cluster capability.
• Map local control relationships to EKS without promoting local evidence into an AWS claim.

The migration dossier must be complete enough for Day 35 to veto production when any boundary is stale.

• [ ] Exact context, source, versions, approval, and healthy precondition recorded
• [ ] Local bypass run seals admitted transport, reviewed Pod UIDs, direct denial, destination-ztunnel observation, and before/after waypoint controls
• [ ] One ztunnel or waypoint failure recovered with old/new UIDs
• [ ] Approved and denied HelixWorks traces replayed with unaffected control
• [ ] HelixWorks-only rollback preserves shared releases and Gateway API CRDs
• [ ] EKS owners, private access, worker capacity, observability, and cost gaps named