Authorize Service-to-Service Traffic at Layer 4

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: scripts/probe-istio-l4-authorization.sh

set -euo pipefail
script_dir="$(cd "$(dirname "$0")" && pwd)"
[ "${MESH_L4_PROBE_APPROVED:-0}" = 1 ] || { echo 'L4 probe veto: set MESH_L4_PROBE_APPROVED=1 after reviewing the exact Pods' >&2; exit 1; }
export MESH_BYPASS_PROBE_APPROVED=1
export MESH_PROBE_STAGE=l4
exec "$script_dir/probe-istio-waypoint-bypass.sh"

Code to reality

Declared intent

Lock the bounded live probe to the L4 stage after explicit review, preventing an accidental waypoint/L7 claim.

Interpreter

The wrapper fixes the stage and delegates to the shared probe, which validates identities, sends and reads HTTP, then requires same-record ztunnel correlation.

Software effect

Control-plane POST remains allowed; evidence-to-generator must return no HTTP bytes and one accepted transport candidate plus correlated policy denial.

Hardware effect

Kindnet and destination ztunnel consume node CPU and packets for independent denied paths, the HTTP attempt, and correlated policy observation.

Observable evidence

Reviewed UIDs, independent CNI timeout, admitted mesh path, zero response bytes, candidate transport result, same-record denial, and before/after POST prove L4.

The source tables below remain the detailed contract. Begin with these customer paths:

• D32-UC-01
  • Person: Enterprise builder
  • Job: Generate and preview an application through the normal Forge control-plane path
  • Observable result: The authenticated control-plane principal reaches generator through ambient L4 policy without a waypoint
• D32-UC-02
  • Person: Platform security operator
  • Job: Prove workload identity is least privilege and independent from customer roles
  • Observable result: Policy names exact SPIFFE-like principals derived from namespace and ServiceAccount

Now turn each customer job into a story with a result that an engineer can check:

• D32-US-01
  • Story: As an enterprise builder, I want the normal Forge request path to remain reachable, so that network hardening does not break generation and preview
  • Observable acceptance: Positive response identifies source ServiceAccount, destination, policy revision, environment, run, and trace
• D32-US-02
  • Story: As a platform security operator, I want unauthorized workload identities rejected, so that a compromised Pod cannot inherit another service’s…
  • Observable acceptance: Denial identifies principal, destination, expected rule, observed result, timestamp, and an unaffected approved request

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D32-FLOW-01
  • Trigger: Builder generates an artifact through the control plane
  • Responsible systems: control-plane ServiceAccount, source/destination ztunnels, Istiod, generator Service
  • Authoritative state: Kubernetes ServiceAccounts and Istio policy objects; Forge stores remain product authorities
  • Owned record: WorkloadIdentityBinding
  • Observable evidence: Actor, source principal, destination, policy generation, response, environment, time, run, and trace
  • Failure signal: Missing identity, non-mTLS connection, wrong principal, or denied approved request
• D32-FLOW-02
  • Trigger: Reviewed evidence Pod sends HTTP GET to the generator Pod IP directly
  • Responsible systems: Kubernetes API, kindnet NetworkPolicy, Istiod distribution, destination ztunnel enforcement, product control, evidence collection
  • Authoritative state: Git policy revision plus immutable Pod/ztunnel UIDs and a fresh local evidence directory
  • Owned record: L4PolicyRevision
  • Observable evidence: Reviewed identities, independent CNI denial, admitted experiment path, no response bytes, candidate transport result plus same-record denial,…
  • Failure signal: Wrong stage, changed identity, CNI ambiguity, any HTTP response byte, unsupported transport result, missing same-record denial, or failed…

Enterprise problem: A private cluster network still lets a compromised workload call every reachable service unless connection authority is explicit, turning one Pod failure into a platform-wide breach. Whole-course context: The incoming enrollment-only overlay gives Forge workloads ambient transport and strict peer authentication but deliberately adds no authorization or waypoint. Today’s slice: We apply ambient-l4-local, whose five destination-specific Layer 4 (L4, the connection layer) policies authorize exact ServiceAccount principals without HTTP interpretation; application tokens and generated-app roles remain separate. End-of-day evidence: Exact overlay, direct control-plane POST, reviewed evidence/generator/broker Pod UIDs, paired NetworkPolicy admission, an HTTP request that returns no response bytes, one same-record destination-ztunnel denial, and before/after L4 positive controls identify the enforcement boundary. Still unsolved: Waypoint selection and HTTP methods/paths require Day 33's separate L7 component.

Reachability without identity policy makes every internal caller equally trusted, so the customer cannot rely on network location as authority.

A policy is not useful if operators cannot connect its principal to the workload that actually made the request.

An allow rule can accidentally block customers or remain unused, so the evidence loop needs both a real product action and a deliberate denial.

IP addresses change with Pods and do not express responsibility, so ambient authorization uses the identity issued to a Kubernetes ServiceAccount over authenticated transport.

Authorization evidence must not become customer business data, because the runtime boundary and generated application have different owners and deletion rules.

Generated-application database: Not created in this slice — existing Forge product stores remain unchanged while Git, Kubernetes, and the evidence store own policy and verification records.

An imprecise example could teach an authority that the repository does not grant, so use one exact destination policy from the L4 component. The other four policies follow the same destination-owned pattern with their required callers.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: generator-l4-boundary
  namespace: zheta-forge
spec:
  selector:
    matchLabels: { app.kubernetes.io/name: generator }
  action: ALLOW
  rules:
    - from: [{ source: { principals: [cluster.local/ns/zheta-forge/sa/control-plane] } }]

This exact source is gitops/apps/forge/mesh/ambient-l4/authorization.yaml. At this stage generator accepts an authenticated connection from the control-plane ServiceAccount; evidence is not allowed, and no waypoint exists. The policy uses a workload selector because ztunnel enforces this L4 boundary directly. Day 33 later patches all five L4 rules to accept only the waypoint principal before adding Service-targeted L7 policy (Istio policy migration).

Policy YAML, rendered manifests, and kubectl get output are static checks. They can prove the intended principal and admission rules exist, but they cannot prove a live direct connection is denied. Apply only the L4 overlay, run its scoped verifier for object state and the allowed POST, then execute the checked-in bounded probe.

Paired allow-bounded-bypass-observation-egress and allow-bounded-bypass-observation-ingress policies admit evidence-to-generator on ports 8080 and 15008, removing NetworkPolicy denial as the explanation for that experiment. A separate broker-to-generator path has no exception and must still return the exact kindnet timeout status 42; this positive proof of CNI enforcement prevents a broken primary network policy engine from being mistaken for mesh evidence.

: "${MESH_CONTEXT:?set the exact kubectl context}"
kubectl --context "$MESH_CONTEXT" apply -k gitops/apps/forge/overlays/ambient-l4-local
MESH_CONTEXT="$MESH_CONTEXT" ./scripts/verify-istio-ambient.sh l4
SOURCE_POD="$(kubectl --context "$MESH_CONTEXT" -n zheta-forge get pods -l app.kubernetes.io/name=evidence -o jsonpath='{.items[0].metadata.name}')"
SOURCE_POD_UID="$(kubectl --context "$MESH_CONTEXT" -n zheta-forge get pod "$SOURCE_POD" -o jsonpath='{.metadata.uid}')"
TARGET_POD="$(kubectl --context "$MESH_CONTEXT" -n zheta-forge get pods -l app.kubernetes.io/name=generator -o jsonpath='{.items[0].metadata.name}')"
TARGET_POD_UID="$(kubectl --context "$MESH_CONTEXT" -n zheta-forge get pod "$TARGET_POD" -o jsonpath='{.metadata.uid}')"
NETWORK_DENY_SOURCE_POD="$(kubectl --context "$MESH_CONTEXT" -n zheta-forge get pods -l app.kubernetes.io/name=broker -o jsonpath='{.items[0].metadata.name}')"
NETWORK_DENY_SOURCE_POD_UID="$(kubectl --context "$MESH_CONTEXT" -n zheta-forge get pod "$NETWORK_DENY_SOURCE_POD" -o jsonpath='{.metadata.uid}')"
printf 'review source=%s/%s target=%s/%s cni-control=%s/%s\n' "$SOURCE_POD" "$SOURCE_POD_UID" "$TARGET_POD" "$TARGET_POD_UID" "$NETWORK_DENY_SOURCE_POD" "$NETWORK_DENY_SOURCE_POD_UID"
MESH_EVIDENCE_DIR="$(mktemp -d "${TMPDIR:-/tmp}/forge-mesh-l4.XXXXXX")"
export SOURCE_POD SOURCE_POD_UID TARGET_POD TARGET_POD_UID NETWORK_DENY_SOURCE_POD NETWORK_DENY_SOURCE_POD_UID MESH_EVIDENCE_DIR
MESH_L4_PROBE_APPROVED=1 ./scripts/probe-istio-l4-authorization.sh

Approval comes only after reviewing the printed names and UIDs and confirming MESH_CONTEXT is the intended local cluster. The wrapper fixes MESH_PROBE_STAGE=l4; the underlying script vetoes a waypoint label, requires generator's control-plane principal, rejects any environment other than local, and refuses ambiguous or changed workloads.

The evidence directory must contain new target.txt, network-policy.yaml, authorization-policy.yaml, denial-stderr.txt, ztunnel-observation.log, ztunnel-denial-record.log, and controls.txt files. controls.txt records successful authenticated POST /generate before and after plus the independent kindnet timeout.

For the admitted mesh experiment, the evidence Pod connects to the exact generator Pod IP, sends GET /healthz, and reads up to 4096 bytes. Any response byte is a fail-open result and aborts the run. Timeout, connection reset, broken pipe, connection abort, and zero-byte EOF are only candidate transport-denial outcomes; an unsupported exception is inconclusive. None of those candidates independently proves Istio authorization.

The probe starts a microsecond-precision UTC window, then writes ztunnel-denial-record.log only when one ztunnel log record contains the exact generator Pod IP and port, spiffe://cluster.local/ns/zheta-forge/sa/evidence, and a policy-denial marker. Finding those fields across different records or accepting a merely nonempty log is insufficient.

Only the joined evidence supports the claim: kindnet independently denies its control path, NetworkPolicy admits the experiment, the HTTP attempt returns no response bytes with one accepted candidate status, one destination-ztunnel record correlates target, identity, and policy, all pinned UIDs stay unchanged, and both L4 POST controls pass. No GET/403 or HTTP policy claim belongs to Day 32.

Ambient enrollment previously authenticated transport without granting destination-specific connection authority. The new L4 stage requires generator's exact control-plane identity and proves that evidence is denied before any waypoint or HTTP policy exists.

L4 authorization answers who may connect, not which HTTP action they may perform.

• Use authenticated ServiceAccount principals instead of Pod IPs as workload authority.
• Keep Kubernetes NetworkPolicy, mesh identity, application tokens, and generated-app roles as separate complementary controls.
• Require independent CNI enforcement, admitted HTTP send/read, zero response bytes, an accepted candidate transport result, same-record destination-ztunnel evidence, and before/after L4 POST controls before claiming enforcement.

The boundary is credible only when identity, policy, denial, and positive control agree.

• [ ] ambient-l4-local applies five L4 policies while waypoint and L7 objects remain absent
• [ ] Reviewed evidence, generator, and broker Pod names and immutable UIDs are recorded
• [ ] Paired NetworkPolicies admit the bounded path on ports 8080 and 15008; independent kindnet control returns status 42
• [ ] Evidence-to-generator sends GET and returns zero response bytes; only timeout/reset/broken-pipe/abort/zero-byte EOF is retained as a candidate
• [ ] One sealed ztunnel record correlates target IP:port, evidence identity, and policy denial in the exact window
• [ ] Authenticated control-plane POST succeeds before and after; no GET/403 claim is made