Install and Enroll Istio Ambient Safely
Install pinned cluster components first, then enroll only the Forge namespace through Git.
Run it in the public monorepo
This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.
Source: gitops/apps/forge/mesh/ambient-enrollment/namespace-patch.yaml
metadata:
name: zheta-forge
labels:
istio.io/dataplane-mode: ambient
Code to reality
- Declared intent
- Enroll Forge in ambient transport without selecting a waypoint or injecting application sidecars.
- Interpreter
- Kustomize patches the Namespace; Istio CNI and ztunnel observe the ambient label while later L4/L7 components remain absent.
- Software effect
- Forge traffic becomes eligible for ambient HBONE transport while application Pod specs and ServiceAccount mappings stay unchanged.
- Hardware effect
- Per-node ztunnel processes consume node CPU, memory, and network capacity outside application containers.
- Observable evidence
- Ambient label, strict peer authentication, compatibility policies, unchanged container identities, and later-stage absence checks prove enrollment state.
Start with the people and the result they need
The source tables below remain the detailed contract. Begin with these customer paths:
- D31-UC-01
- Person: Platform operator
- Job: Install approved ambient prerequisites into one exact cluster
- Observable result: Gateway API CRD, istio-base, istiod, istio-cni, and ztunnel are present at pinned versions while Forge remains unenrolled
- D31-UC-02
- Person: Forge service owner
- Job: Enroll only zheta-forge without changing application containers or introducing a waypoint
- Observable result: The ambient label and forge-strict-mtls reconcile; HBONE and kubelet-health paths remain declared; five workloads retain existing ServiceAccounts
Turn each customer job into a testable story
Now turn each customer job into a story with a result that an engineer can check:
- D31-US-01
- Story: As a platform operator, I want pinned ambient components installed behind context and approval gates, so that an accidental cluster or version…
- Observable acceptance: Record contains context, cluster-info result, approval, Gateway API URL and checksum, Helm chart versions, rollout identities, time, and run ID
- D31-US-02
- Story: As a Forge service owner, I want Git-controlled namespace enrollment with strict peer authentication, so that I can reverse only this namespace…
- Observable acceptance: Ambient label and STRICT mode equal source; no waypoint label or AuthorizationPolicy exists; five ServiceAccounts persist
Add real state and observable proof
Finally trace each story through the system that owns its state and the evidence that proves the outcome:
- D31-FLOW-01
- Trigger: Approved operator selects a disposable cluster and service owner approves namespace enrollment
- Responsible systems: Shell gates, curl, checksum tool, Kubernetes API server, Helm, Istio charts, Deployment and DaemonSet controllers
- Authoritative state: mesh/istio-ambient/versions.env, install script, Helm release records, and Kubernetes API objects
- Owned record: MeshVersionPin
- Observable evidence: Context, approval, source revision, checksums, Helm releases, rollout UIDs, enrollment objects, absence checks, environment, time, and run ID
- Failure signal: Context or checksum veto, non-deployed Helm release, unavailable CRD, or failed istiod, CNI, or ztunnel rollout
- D31-FLOW-02
- Trigger: Context, approval, checksum, rollout, or post-enrollment probe disagrees
- Responsible systems: Kustomize, Kubernetes API server, Istio controllers, Istio CNI, ztunnel, and Forge Deployment controllers
- Authoritative state: Git overlay is desired state; Namespace, PeerAuthentication, NetworkPolicies, ServiceAccounts, Pods, and evidence record are observed state
- Owned record: MeshInstallation
- Observable evidence: Veto reason, last completed gate, unchanged or recovered baseline result, affected namespace, context, source revision, time, and incident ID
- Failure signal: Label/strict-mode drift, missing compatibility policy, unexpected waypoint/authorization state, or workload identity change
The enterprise problem and today’s slice
Enterprise problem: Installing a mesh and enrolling workloads as one opaque action can redirect live packets before the cluster components, target context, or rollback boundary are proven, turning a security change into an outage. Whole-course context: The incoming Day 30 artifact is a revision-bound live baseline of the five-service Forge request path, including checked-in positive and denied NetworkPolicy evidence. Today’s slice: The platform operator installs checksum-pinned Gateway API v1.5.1 resources and Helm-pinned Istio 1.30.3 components; the Forge owner then applies only ambient-enrollment-local, which adds ambient enrollment, namespace-wide STRICT peer authentication, and HBONE/health-probe NetworkPolicy compatibility. End-of-day evidence: Exact context, approval, versions, Helm releases, controller and node-agent rollouts, the ambient label, strict mode, compatibility policies, and unchanged workload identities form the enrollment record. Still unsolved: L4 authorization, waypoint selection, L7 method/path policy, live denial, mutual-TLS telemetry, bounded failure, rollback execution, customer ingress, and production rollout remain deferred.
Customer use cases
A cluster can support ambient mode without enrolling every namespace. Separating installation from enrollment keeps the platform boundary and application boundary independently reviewable and reversible.
| Use case ID | Actor | Customer job | Success outcome | Denial or recovery evidence |
|---|---|---|---|---|
| D31-UC-01 | Platform operator | Install approved ambient prerequisites into one exact cluster | Gateway API CRD, istio-base, istiod, istio-cni, and ztunnel are present at pinned versions while Forge remains unenrolled | Wrong context, absent approval, download checksum mismatch, failed rollout, or version mismatch vetoes enrollment |
| D31-UC-02 | Forge service owner | Enroll only zheta-forge without changing application containers or introducing a waypoint | The ambient label and forge-strict-mtls reconcile; HBONE and kubelet-health paths remain declared; five workloads retain existing ServiceAccounts | Missing label/policy, unexpected waypoint label, changed identity, or failed component readiness triggers stop and rollback review |
Actor-centred user stories
Component readiness and customer traffic are different claims, so neither can replace the other. These stories require both platform evidence and application-owner evidence.
| Story ID | Use case IDs | User story | Observable acceptance conditions |
|---|---|---|---|
| D31-US-01 | D31-UC-01 | As a platform operator, I want pinned ambient components installed behind context and approval gates, so that an accidental cluster or version cannot be changed | Record contains context, cluster-info result, approval, Gateway API URL and checksum, Helm chart versions, rollout identities, time, and run ID |
| D31-US-02 | D31-UC-02 | As a Forge service owner, I want Git-controlled namespace enrollment with strict peer authentication, so that I can reverse only this namespace before adding authorization policy | Ambient label and STRICT mode equal source; no waypoint label or AuthorizationPolicy exists; five ServiceAccounts persist |
End-to-end product flows
Enrollment must begin with the operator naming the target, not with Helm changing it. Istio’s Helm guide installs base resources, Gateway API resources, istiod, the CNI node agent, and the ztunnel DaemonSet as separate components (Istio Helm installation).
| Flow ID | Use case IDs | Path | Trigger | Numbered steps | Terminal evidence |
|---|---|---|---|---|---|
| D31-FLOW-01 | D31-UC-01, D31-UC-02 | Happy | Approved operator selects a disposable cluster and service owner approves namespace enrollment | 1. Verify exact current context and cluster reachability. 2. Read pinned versions. 3. Download and checksum Gateway API bundle. 4. Server-side apply it. 5. Install four pinned Helm releases. 6. Verify control-plane and node components. 7. Review and apply ambient-enrollment-local. 8. Verify ambient label, strict mode, compatibility policies, identities, and absence of waypoint/L4/L7 state. | Context, approval, source revision, checksums, Helm releases, rollout UIDs, enrollment objects, absence checks, environment, time, and run ID |
| D31-FLOW-02 | D31-UC-01, D31-UC-02 | Denied | Context, approval, checksum, rollout, or post-enrollment probe disagrees | 1. Stop at the first failed gate. 2. Record expected and observed values. 3. Make no broader namespace change. 4. If enrollment occurred, preserve evidence and select the repository rollback procedure. 5. Re-run the pre-mesh positive control. | Veto reason, last completed gate, unchanged or recovered baseline result, affected namespace, context, source revision, time, and incident ID |
The minimal system has two accountable actions. The operator prepares cluster capability; the Forge owner chooses whether one namespace uses it; the unchanged customer probes decide whether the slice is acceptable.
System design derived from the flows
The risk is hidden ownership: Helm does not own the Forge namespace, and the Forge overlay does not own cluster-scoped Istio prerequisites. Ambient ztunnel is a per-node Layer 4 proxy, while a waypoint is an optional Layer 7 proxy (Istio ambient overview).
| Use case ID | Entry point | Responsible services | Authoritative store | Failure evidence |
|---|---|---|---|---|
| D31-UC-01 | scripts/install-istio-ambient.sh with MESH_CONTEXT and MESH_INSTALL_APPROVED=1 | Shell gates, curl, checksum tool, Kubernetes API server, Helm, Istio charts, Deployment and DaemonSet controllers | mesh/istio-ambient/versions.env, install script, Helm release records, and Kubernetes API objects | Context or checksum veto, non-deployed Helm release, unavailable CRD, or failed istiod, CNI, or ztunnel rollout |
| D31-UC-02 | Reviewed kubectl apply -k gitops/apps/forge/overlays/ambient-enrollment-local in the exact context | Kustomize, Kubernetes API server, Istio controllers, Istio CNI, ztunnel, and Forge Deployment controllers | Git overlay is desired state; Namespace, PeerAuthentication, NetworkPolicies, ServiceAccounts, Pods, and evidence record are observed state | Label/strict-mode drift, missing compatibility policy, unexpected waypoint/authorization state, or workload identity change |
The original operator-to-owner path stays unchanged. The new boxes reveal the cluster control plane, node processes, and Git-owned enrollment objects that consume node CPU, memory, disk, and network while application containers remain unchanged.
Data model and ownership
An installation record cannot authorize namespace enrollment, and a namespace label cannot prove the installed versions. Correlate the records, but keep their owners and rollback lifecycles separate.
Generated-application database: Not created in this slice — cluster and namespace desired state plus immutable enrollment evidence are sufficient; generated applications keep ownership of their tenant and domain records.
| Record or entity | Store and owner | Primary key | Foreign key or opaque reference | Tenant key | Material constraint | Lifecycle and deletion | Use case IDs |
|---|---|---|---|---|---|---|---|
| MeshVersionPin | Git versions.env, owned by platform engineering | source revision plus component name | Opaque upstream release URL and checksum reference | None — one approved pin set is cluster-wide | Istio is exactly 1.30.3; Gateway API is exactly v1.5.1; downloaded bundle matches the stored SHA-256 | Review and promote in Git, retain history, replace only through upgrade review | D31-UC-01 |
| MeshInstallation | Helm records and Kubernetes API, owned by platform engineering | cluster identity plus release name | Opaque version-pin and install-run references | None — cluster prerequisite is shared | Four named releases use the pin; required CRD and rollouts are ready in the exact context | Install or upgrade independently, verify, retain release history, uninstall only after namespaces leave mesh | D31-UC-01, D31-UC-02 |
| ForgeAmbientEnrollment | Git overlay and Namespace object, owned by Forge team | cluster identity plus namespace zheta-forge | Opaque mesh-installation revision and overlay revision references | zheta-forge namespace — no generated-app tenant authority transfers | Ambient label equals source; waypoint label and authorization objects remain absent; existing ServiceAccount mapping remains unchanged | Approve, reconcile, verify, remove component through reviewed rollback, retain Git history | D31-UC-02 |
| ForgePeerAuthentication | Git manifest and Istio configuration API, owned by Forge security team | namespace plus forge-strict-mtls | Opaque enrollment revision reference | zheta-forge namespace — applies to workloads in that namespace | Namespace-wide spec.mtls.mode is STRICT with no selector exception | Reconcile with enrollment, verify mode, remove through rollback review, retain policy history | D31-UC-02 |
| AmbientEnrollmentRun | Evidence artifact store, owned by platform operations | run_id | Opaque context, approval, pin, Helm, object UID, absence-check, and source-revision references | environment plus zheta-forge namespace | One evidence window contains platform readiness and exact enrollment desired/observed state; it makes no live authorization claim | Create, seal, compare during later policy work, retain through rollback window, expire by evidence policy | D31-UC-01, D31-UC-02 |
The final view adds durable version, installation, enrollment, policy, and evidence records. Each can be reviewed or rolled back by its owner without pretending that one green object proves the others.
Pin and install the cluster boundary
Downloading a mutable remote manifest or using an unpinned Helm chart can make two operators install different systems from the same instructions. The implementation fixes that problem with this exact mesh/istio-ambient/versions.env content.
ISTIO_VERSION=1.30.3
GATEWAY_API_VERSION=v1.5.1
GATEWAY_API_SHA256=64ec76609a6ac885e0405dea79ca509c229fa019d342f0857aa8b6bdc8b8ba92
| Effect field | What happens |
|---|---|
| Declared intent | Select exact Istio charts and one checksum-verified Gateway API experimental bundle |
| Interpreter | istio-ambient-lib.sh, install shell script, curl, shasum, Helm, and Kubernetes API server |
| Software effect | The server stores Gateway API resources and Helm installs istio-base, istiod, istio-cni, and ztunnel at Istio 1.30.3 |
| Hardware effect | Control-plane Pods and per-node DaemonSet Pods consume cluster CPU, memory, disk, and network |
| Evidence | Stored checksum equals downloaded bytes; Helm releases report deployed; CRD and rollout checks succeed in the exact context |
The official Istio Helm guide currently shows Gateway API v1.5.1 experimental resources and separate ambient components, matching the repository’s pinned sequence (Istio Helm installation). The repository is authoritative for this lab’s exact pins; “latest” is not an accepted input.
export MESH_CONTEXT="${MESH_CONTEXT:?set the exact disposable cluster context}"
export MESH_INSTALL_APPROVED=1
./scripts/install-istio-ambient.sh
./scripts/verify-istio-ambient.sh control-plane
The script refuses to continue unless the shell’s current context equals MESH_CONTEXT, the API answers, the explicit approval equals one, and the downloaded SHA-256 matches. Only then do the API server and Helm controllers change software state. A successful exit proves named component presence and rollout; it does not yet prove Forge traffic or enrollment.
Enroll only the Forge namespace
Installing ambient-capable components does not automatically redirect Forge traffic. Istio documents namespace enrollment through istio.io/dataplane-mode=ambient; unlabeled namespaces remain out of the ambient data plane (enable ambient mode).
These are the exact enrollment and peer-authentication objects from mesh/ambient-enrollment/namespace-patch.yaml and peer-authentication.yaml:
apiVersion: v1
kind: Namespace
metadata:
name: zheta-forge
labels:
istio.io/dataplane-mode: ambient
---
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: forge-strict-mtls
namespace: zheta-forge
spec:
mtls:
mode: STRICT
| Effect field | What happens |
|---|---|
| Declared intent | Enroll zheta-forge in ambient transport and require tunneled mutual TLS without selecting a waypoint or adding authorization |
| Interpreter | Kustomize, Kubernetes API server, Istio CNI, ztunnel, and Istiod |
| Software effect | The Namespace gains only the ambient label and the Istio configuration store gains forge-strict-mtls; application Pod specs gain no sidecar container |
| Hardware effect | Existing node agents intercept enrolled traffic; proxies and control plane consume node CPU, memory, and network |
| Evidence | Live ambient label and spec.mtls.mode equal source, HBONE/health policy objects exist, waypoint and authorization state remain absent, and Pods retain one application container and same-named ServiceAccounts |
In ambient mode, STRICT is useful for ensuring that connections cannot bypass the mesh; DISABLE is not supported (Istio PeerAuthentication). This object proves declared policy, not observed encryption. Packet or ztunnel telemetry proof belongs to a later evidence gate.
kubectl kustomize gitops/apps/forge/overlays/ambient-enrollment-local >/tmp/forge-ambient-enrollment-local.yaml
./scripts/verify-ambient-source.sh
kubectl --context "${MESH_CONTEXT:?set the exact disposable cluster context}" apply -k gitops/apps/forge/overlays/ambient-enrollment-local
MESH_CONTEXT="$MESH_CONTEXT" ./scripts/verify-istio-ambient.sh enrollment
kubectl --context "$MESH_CONTEXT" -n zheta-forge get pod -o custom-columns='NAME:.metadata.name,CONTAINERS:.spec.containers[*].name,SERVICE_ACCOUNT:.spec.serviceAccountName'
The source verifier renders all three stages and checks their ordering before mutation. This explicit apply reconciles only enrollment. The scoped live verifier checks ambient label, strict mode, HBONE/health NetworkPolicies, and cluster components; it deliberately does not claim a live L4 denial, waypoint, or HTTP policy. Day 30 remains the last sealed customer-traffic baseline until Day 32 adds and probes L4 authorization.
Failure boundary and decision rule
Partial installation is not permission to enroll, and successful enrollment is not permission to continue after a customer regression. Diagnose from the lowest changed layer upward.
| Observation | Interpretation | Action |
|---|---|---|
| Context or approval gate fails | Target authority is absent | Change nothing; correct the reviewed input |
| Gateway API checksum differs | Downloaded bytes are not the approved artifact | Change nothing; investigate release URL or pin through review |
istiod, CNI, or ztunnel rollout fails | Cluster capability is incomplete | Keep Forge unenrolled and inspect the failing owner |
| Namespace label or strict mode differs | Git desired state has not reconciled | Stop application testing and inspect the deployment controller |
| Waypoint label or AuthorizationPolicy appears | A later stage leaked into enrollment | Stop and restore the enrollment-only overlay |
| Enrollment objects agree but no live traffic proof exists | This verifier is intentionally structural | Do not infer authorization; proceed only to Day 32's bounded live probe |
Decision rule: install cluster capability first, prove it, enroll one namespace second, and refuse to add L4 or L7 policy until the enrollment-only verifier agrees with Git. Istio’s migration guidance likewise separates ambient-component installation from adding the namespace label (install ambient components).
Before and after, side by side
Before this day, Forge had a measured Kubernetes-only path but no mesh workload transport. Afterward, the same containers and ServiceAccounts are enrolled through one Git label, node ztunnels can carry their traffic, strict peer authentication is declared, and later waypoint/L4/L7 objects remain absent. This structural enrollment evidence does not claim live authorization or mutual-TLS telemetry.
Key takeaways
Treating installation and enrollment as one command hides which owner must stop or recover a failure. Ambient enrollment is a controlled ownership handoff.
- Pin remote bytes and Helm charts; verify context and explicit approval before mutation.
- Installing ambient support and enrolling
zheta-forgeare separate decisions owned by different teams. - The ambient label selects ztunnel transport; waypoint selection belongs to Day 33.
STRICTdeclares required tunneled mutual TLS, while live authorization evidence belongs to the next stages.- Keep the change only when components, source, enrollment state, absence checks, and workload identities agree.
Checklist
Day 31 is complete only when the cluster and customer evidence come from the same reviewed window.
- [ ]
MESH_CONTEXTequals the current reachable disposable context - [ ] Gateway API
v1.5.1bundle matches the pinned SHA-256 - [ ] Four Istio
1.30.3Helm releases and their rollouts are ready - [ ]
ambient-enrollment-localrenders and applies without L4, waypoint, or L7 objects - [ ]
zheta-forgehas only the ambient mesh label andforge-strict-mtlsisSTRICT - [ ] Five application containers retain their same-named ServiceAccounts
- [ ] Enrollment verifier passes without claiming live authorization evidence