31

Install and Enroll Istio Ambient Safely

Install pinned cluster components first, then enroll only the Forge namespace through Git.

Run it in the public monorepo

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: gitops/apps/forge/mesh/ambient-enrollment/namespace-patch.yaml

metadata:
  name: zheta-forge
  labels:
    istio.io/dataplane-mode: ambient

Code to reality

Declared intent
Enroll Forge in ambient transport without selecting a waypoint or injecting application sidecars.
Interpreter
Kustomize patches the Namespace; Istio CNI and ztunnel observe the ambient label while later L4/L7 components remain absent.
Software effect
Forge traffic becomes eligible for ambient HBONE transport while application Pod specs and ServiceAccount mappings stay unchanged.
Hardware effect
Per-node ztunnel processes consume node CPU, memory, and network capacity outside application containers.
Observable evidence
Ambient label, strict peer authentication, compatibility policies, unchanged container identities, and later-stage absence checks prove enrollment state.

Start with the people and the result they need

The source tables below remain the detailed contract. Begin with these customer paths:

  • D31-UC-01
    • Person: Platform operator
    • Job: Install approved ambient prerequisites into one exact cluster
    • Observable result: Gateway API CRD, istio-base, istiod, istio-cni, and ztunnel are present at pinned versions while Forge remains unenrolled
  • D31-UC-02
    • Person: Forge service owner
    • Job: Enroll only zheta-forge without changing application containers or introducing a waypoint
    • Observable result: The ambient label and forge-strict-mtls reconcile; HBONE and kubelet-health paths remain declared; five workloads retain existing ServiceAccounts

Turn each customer job into a testable story

Now turn each customer job into a story with a result that an engineer can check:

  • D31-US-01
    • Story: As a platform operator, I want pinned ambient components installed behind context and approval gates, so that an accidental cluster or version…
    • Observable acceptance: Record contains context, cluster-info result, approval, Gateway API URL and checksum, Helm chart versions, rollout identities, time, and run ID
  • D31-US-02
    • Story: As a Forge service owner, I want Git-controlled namespace enrollment with strict peer authentication, so that I can reverse only this namespace…
    • Observable acceptance: Ambient label and STRICT mode equal source; no waypoint label or AuthorizationPolicy exists; five ServiceAccounts persist

Add real state and observable proof

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

  • D31-FLOW-01
    • Trigger: Approved operator selects a disposable cluster and service owner approves namespace enrollment
    • Responsible systems: Shell gates, curl, checksum tool, Kubernetes API server, Helm, Istio charts, Deployment and DaemonSet controllers
    • Authoritative state: mesh/istio-ambient/versions.env, install script, Helm release records, and Kubernetes API objects
    • Owned record: MeshVersionPin
    • Observable evidence: Context, approval, source revision, checksums, Helm releases, rollout UIDs, enrollment objects, absence checks, environment, time, and run ID
    • Failure signal: Context or checksum veto, non-deployed Helm release, unavailable CRD, or failed istiod, CNI, or ztunnel rollout
  • D31-FLOW-02
    • Trigger: Context, approval, checksum, rollout, or post-enrollment probe disagrees
    • Responsible systems: Kustomize, Kubernetes API server, Istio controllers, Istio CNI, ztunnel, and Forge Deployment controllers
    • Authoritative state: Git overlay is desired state; Namespace, PeerAuthentication, NetworkPolicies, ServiceAccounts, Pods, and evidence record are observed state
    • Owned record: MeshInstallation
    • Observable evidence: Veto reason, last completed gate, unchanged or recovered baseline result, affected namespace, context, source revision, time, and incident ID
    • Failure signal: Label/strict-mode drift, missing compatibility policy, unexpected waypoint/authorization state, or workload identity change

The enterprise problem and today’s slice

Enterprise problem: Installing a mesh and enrolling workloads as one opaque action can redirect live packets before the cluster components, target context, or rollback boundary are proven, turning a security change into an outage. Whole-course context: The incoming Day 30 artifact is a revision-bound live baseline of the five-service Forge request path, including checked-in positive and denied NetworkPolicy evidence. Today’s slice: The platform operator installs checksum-pinned Gateway API v1.5.1 resources and Helm-pinned Istio 1.30.3 components; the Forge owner then applies only ambient-enrollment-local, which adds ambient enrollment, namespace-wide STRICT peer authentication, and HBONE/health-probe NetworkPolicy compatibility. End-of-day evidence: Exact context, approval, versions, Helm releases, controller and node-agent rollouts, the ambient label, strict mode, compatibility policies, and unchanged workload identities form the enrollment record. Still unsolved: L4 authorization, waypoint selection, L7 method/path policy, live denial, mutual-TLS telemetry, bounded failure, rollback execution, customer ingress, and production rollout remain deferred.

Customer use cases

A cluster can support ambient mode without enrolling every namespace. Separating installation from enrollment keeps the platform boundary and application boundary independently reviewable and reversible.

Use case IDActorCustomer jobSuccess outcomeDenial or recovery evidence
D31-UC-01Platform operatorInstall approved ambient prerequisites into one exact clusterGateway API CRD, istio-base, istiod, istio-cni, and ztunnel are present at pinned versions while Forge remains unenrolledWrong context, absent approval, download checksum mismatch, failed rollout, or version mismatch vetoes enrollment
D31-UC-02Forge service ownerEnroll only zheta-forge without changing application containers or introducing a waypointThe ambient label and forge-strict-mtls reconcile; HBONE and kubelet-health paths remain declared; five workloads retain existing ServiceAccountsMissing label/policy, unexpected waypoint label, changed identity, or failed component readiness triggers stop and rollback review

Actor-centred user stories

Component readiness and customer traffic are different claims, so neither can replace the other. These stories require both platform evidence and application-owner evidence.

Story IDUse case IDsUser storyObservable acceptance conditions
D31-US-01D31-UC-01As a platform operator, I want pinned ambient components installed behind context and approval gates, so that an accidental cluster or version cannot be changedRecord contains context, cluster-info result, approval, Gateway API URL and checksum, Helm chart versions, rollout identities, time, and run ID
D31-US-02D31-UC-02As a Forge service owner, I want Git-controlled namespace enrollment with strict peer authentication, so that I can reverse only this namespace before adding authorization policyAmbient label and STRICT mode equal source; no waypoint label or AuthorizationPolicy exists; five ServiceAccounts persist

End-to-end product flows

Enrollment must begin with the operator naming the target, not with Helm changing it. Istio’s Helm guide installs base resources, Gateway API resources, istiod, the CNI node agent, and the ztunnel DaemonSet as separate components (Istio Helm installation).

Flow IDUse case IDsPathTriggerNumbered stepsTerminal evidence
D31-FLOW-01D31-UC-01, D31-UC-02HappyApproved operator selects a disposable cluster and service owner approves namespace enrollment1. Verify exact current context and cluster reachability. 2. Read pinned versions. 3. Download and checksum Gateway API bundle. 4. Server-side apply it. 5. Install four pinned Helm releases. 6. Verify control-plane and node components. 7. Review and apply ambient-enrollment-local. 8. Verify ambient label, strict mode, compatibility policies, identities, and absence of waypoint/L4/L7 state.Context, approval, source revision, checksums, Helm releases, rollout UIDs, enrollment objects, absence checks, environment, time, and run ID
D31-FLOW-02D31-UC-01, D31-UC-02DeniedContext, approval, checksum, rollout, or post-enrollment probe disagrees1. Stop at the first failed gate. 2. Record expected and observed values. 3. Make no broader namespace change. 4. If enrollment occurred, preserve evidence and select the repository rollback procedure. 5. Re-run the pre-mesh positive control.Veto reason, last completed gate, unchanged or recovered baseline result, affected namespace, context, source revision, time, and incident ID

The minimal system has two accountable actions. The operator prepares cluster capability; the Forge owner chooses whether one namespace uses it; the unchanged customer probes decide whether the slice is acceptable.

System design derived from the flows

The risk is hidden ownership: Helm does not own the Forge namespace, and the Forge overlay does not own cluster-scoped Istio prerequisites. Ambient ztunnel is a per-node Layer 4 proxy, while a waypoint is an optional Layer 7 proxy (Istio ambient overview).

Use case IDEntry pointResponsible servicesAuthoritative storeFailure evidence
D31-UC-01scripts/install-istio-ambient.sh with MESH_CONTEXT and MESH_INSTALL_APPROVED=1Shell gates, curl, checksum tool, Kubernetes API server, Helm, Istio charts, Deployment and DaemonSet controllersmesh/istio-ambient/versions.env, install script, Helm release records, and Kubernetes API objectsContext or checksum veto, non-deployed Helm release, unavailable CRD, or failed istiod, CNI, or ztunnel rollout
D31-UC-02Reviewed kubectl apply -k gitops/apps/forge/overlays/ambient-enrollment-local in the exact contextKustomize, Kubernetes API server, Istio controllers, Istio CNI, ztunnel, and Forge Deployment controllersGit overlay is desired state; Namespace, PeerAuthentication, NetworkPolicies, ServiceAccounts, Pods, and evidence record are observed stateLabel/strict-mode drift, missing compatibility policy, unexpected waypoint/authorization state, or workload identity change

The original operator-to-owner path stays unchanged. The new boxes reveal the cluster control plane, node processes, and Git-owned enrollment objects that consume node CPU, memory, disk, and network while application containers remain unchanged.

Data model and ownership

An installation record cannot authorize namespace enrollment, and a namespace label cannot prove the installed versions. Correlate the records, but keep their owners and rollback lifecycles separate.

Generated-application database: Not created in this slice — cluster and namespace desired state plus immutable enrollment evidence are sufficient; generated applications keep ownership of their tenant and domain records.

Record or entityStore and ownerPrimary keyForeign key or opaque referenceTenant keyMaterial constraintLifecycle and deletionUse case IDs
MeshVersionPinGit versions.env, owned by platform engineeringsource revision plus component nameOpaque upstream release URL and checksum referenceNone — one approved pin set is cluster-wideIstio is exactly 1.30.3; Gateway API is exactly v1.5.1; downloaded bundle matches the stored SHA-256Review and promote in Git, retain history, replace only through upgrade reviewD31-UC-01
MeshInstallationHelm records and Kubernetes API, owned by platform engineeringcluster identity plus release nameOpaque version-pin and install-run referencesNone — cluster prerequisite is sharedFour named releases use the pin; required CRD and rollouts are ready in the exact contextInstall or upgrade independently, verify, retain release history, uninstall only after namespaces leave meshD31-UC-01, D31-UC-02
ForgeAmbientEnrollmentGit overlay and Namespace object, owned by Forge teamcluster identity plus namespace zheta-forgeOpaque mesh-installation revision and overlay revision referenceszheta-forge namespace — no generated-app tenant authority transfersAmbient label equals source; waypoint label and authorization objects remain absent; existing ServiceAccount mapping remains unchangedApprove, reconcile, verify, remove component through reviewed rollback, retain Git historyD31-UC-02
ForgePeerAuthenticationGit manifest and Istio configuration API, owned by Forge security teamnamespace plus forge-strict-mtlsOpaque enrollment revision referencezheta-forge namespace — applies to workloads in that namespaceNamespace-wide spec.mtls.mode is STRICT with no selector exceptionReconcile with enrollment, verify mode, remove through rollback review, retain policy historyD31-UC-02
AmbientEnrollmentRunEvidence artifact store, owned by platform operationsrun_idOpaque context, approval, pin, Helm, object UID, absence-check, and source-revision referencesenvironment plus zheta-forge namespaceOne evidence window contains platform readiness and exact enrollment desired/observed state; it makes no live authorization claimCreate, seal, compare during later policy work, retain through rollback window, expire by evidence policyD31-UC-01, D31-UC-02

The final view adds durable version, installation, enrollment, policy, and evidence records. Each can be reviewed or rolled back by its owner without pretending that one green object proves the others.

Pin and install the cluster boundary

Downloading a mutable remote manifest or using an unpinned Helm chart can make two operators install different systems from the same instructions. The implementation fixes that problem with this exact mesh/istio-ambient/versions.env content.

ISTIO_VERSION=1.30.3
GATEWAY_API_VERSION=v1.5.1
GATEWAY_API_SHA256=64ec76609a6ac885e0405dea79ca509c229fa019d342f0857aa8b6bdc8b8ba92
Effect fieldWhat happens
Declared intentSelect exact Istio charts and one checksum-verified Gateway API experimental bundle
Interpreteristio-ambient-lib.sh, install shell script, curl, shasum, Helm, and Kubernetes API server
Software effectThe server stores Gateway API resources and Helm installs istio-base, istiod, istio-cni, and ztunnel at Istio 1.30.3
Hardware effectControl-plane Pods and per-node DaemonSet Pods consume cluster CPU, memory, disk, and network
EvidenceStored checksum equals downloaded bytes; Helm releases report deployed; CRD and rollout checks succeed in the exact context

The official Istio Helm guide currently shows Gateway API v1.5.1 experimental resources and separate ambient components, matching the repository’s pinned sequence (Istio Helm installation). The repository is authoritative for this lab’s exact pins; “latest” is not an accepted input.

export MESH_CONTEXT="${MESH_CONTEXT:?set the exact disposable cluster context}"
export MESH_INSTALL_APPROVED=1
./scripts/install-istio-ambient.sh
./scripts/verify-istio-ambient.sh control-plane

The script refuses to continue unless the shell’s current context equals MESH_CONTEXT, the API answers, the explicit approval equals one, and the downloaded SHA-256 matches. Only then do the API server and Helm controllers change software state. A successful exit proves named component presence and rollout; it does not yet prove Forge traffic or enrollment.

Enroll only the Forge namespace

Installing ambient-capable components does not automatically redirect Forge traffic. Istio documents namespace enrollment through istio.io/dataplane-mode=ambient; unlabeled namespaces remain out of the ambient data plane (enable ambient mode).

These are the exact enrollment and peer-authentication objects from mesh/ambient-enrollment/namespace-patch.yaml and peer-authentication.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: zheta-forge
  labels:
    istio.io/dataplane-mode: ambient
---
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: forge-strict-mtls
  namespace: zheta-forge
spec:
  mtls:
    mode: STRICT
Effect fieldWhat happens
Declared intentEnroll zheta-forge in ambient transport and require tunneled mutual TLS without selecting a waypoint or adding authorization
InterpreterKustomize, Kubernetes API server, Istio CNI, ztunnel, and Istiod
Software effectThe Namespace gains only the ambient label and the Istio configuration store gains forge-strict-mtls; application Pod specs gain no sidecar container
Hardware effectExisting node agents intercept enrolled traffic; proxies and control plane consume node CPU, memory, and network
EvidenceLive ambient label and spec.mtls.mode equal source, HBONE/health policy objects exist, waypoint and authorization state remain absent, and Pods retain one application container and same-named ServiceAccounts

In ambient mode, STRICT is useful for ensuring that connections cannot bypass the mesh; DISABLE is not supported (Istio PeerAuthentication). This object proves declared policy, not observed encryption. Packet or ztunnel telemetry proof belongs to a later evidence gate.

kubectl kustomize gitops/apps/forge/overlays/ambient-enrollment-local >/tmp/forge-ambient-enrollment-local.yaml
./scripts/verify-ambient-source.sh
kubectl --context "${MESH_CONTEXT:?set the exact disposable cluster context}" apply -k gitops/apps/forge/overlays/ambient-enrollment-local
MESH_CONTEXT="$MESH_CONTEXT" ./scripts/verify-istio-ambient.sh enrollment
kubectl --context "$MESH_CONTEXT" -n zheta-forge get pod -o custom-columns='NAME:.metadata.name,CONTAINERS:.spec.containers[*].name,SERVICE_ACCOUNT:.spec.serviceAccountName'

The source verifier renders all three stages and checks their ordering before mutation. This explicit apply reconciles only enrollment. The scoped live verifier checks ambient label, strict mode, HBONE/health NetworkPolicies, and cluster components; it deliberately does not claim a live L4 denial, waypoint, or HTTP policy. Day 30 remains the last sealed customer-traffic baseline until Day 32 adds and probes L4 authorization.

Failure boundary and decision rule

Partial installation is not permission to enroll, and successful enrollment is not permission to continue after a customer regression. Diagnose from the lowest changed layer upward.

ObservationInterpretationAction
Context or approval gate failsTarget authority is absentChange nothing; correct the reviewed input
Gateway API checksum differsDownloaded bytes are not the approved artifactChange nothing; investigate release URL or pin through review
istiod, CNI, or ztunnel rollout failsCluster capability is incompleteKeep Forge unenrolled and inspect the failing owner
Namespace label or strict mode differsGit desired state has not reconciledStop application testing and inspect the deployment controller
Waypoint label or AuthorizationPolicy appearsA later stage leaked into enrollmentStop and restore the enrollment-only overlay
Enrollment objects agree but no live traffic proof existsThis verifier is intentionally structuralDo not infer authorization; proceed only to Day 32's bounded live probe

Decision rule: install cluster capability first, prove it, enroll one namespace second, and refuse to add L4 or L7 policy until the enrollment-only verifier agrees with Git. Istio’s migration guidance likewise separates ambient-component installation from adding the namespace label (install ambient components).

Before and after, side by side

Before this day, Forge had a measured Kubernetes-only path but no mesh workload transport. Afterward, the same containers and ServiceAccounts are enrolled through one Git label, node ztunnels can carry their traffic, strict peer authentication is declared, and later waypoint/L4/L7 objects remain absent. This structural enrollment evidence does not claim live authorization or mutual-TLS telemetry.

Key takeaways

Treating installation and enrollment as one command hides which owner must stop or recover a failure. Ambient enrollment is a controlled ownership handoff.

  • Pin remote bytes and Helm charts; verify context and explicit approval before mutation.
  • Installing ambient support and enrolling zheta-forge are separate decisions owned by different teams.
  • The ambient label selects ztunnel transport; waypoint selection belongs to Day 33.
  • STRICT declares required tunneled mutual TLS, while live authorization evidence belongs to the next stages.
  • Keep the change only when components, source, enrollment state, absence checks, and workload identities agree.

Checklist

Day 31 is complete only when the cluster and customer evidence come from the same reviewed window.

  • [ ] MESH_CONTEXT equals the current reachable disposable context
  • [ ] Gateway API v1.5.1 bundle matches the pinned SHA-256
  • [ ] Four Istio 1.30.3 Helm releases and their rollouts are ready
  • [ ] ambient-enrollment-local renders and applies without L4, waypoint, or L7 objects
  • [ ] zheta-forge has only the ambient mesh label and forge-strict-mtls is STRICT
  • [ ] Five application containers retain their same-named ServiceAccounts
  • [ ] Enrollment verifier passes without claiming live authorization evidence