Authorize Service-to-Service Traffic at Layer 4

Enterprise problem: A private cluster network still lets a compromised workload call every reachable service unless connection authority is explicit, turning one Pod failure into a platform-wide breach. Whole-course context: The incoming enrollment-only overlay gives HelixWorks workloads ambient transport and strict peer authentication but deliberately adds no authorization or waypoint. Today’s slice: We apply ambient-l4-local, whose five destination-specific Layer 4 (L4, the connection layer) policies authorize exact ServiceAccount principals without HTTP interpretation; application tokens and generated-app roles remain separate. End-of-day evidence: Exact overlay, direct control-plane POST, reviewed evidence/generator/broker Pod UIDs, paired NetworkPolicy admission, an HTTP request that returns no response bytes, one same-record destination-ztunnel denial, and before/after L4 positive controls identify the enforcement boundary. Still unsolved: Waypoint selection and HTTP methods/paths require Day 33's separate L7 component.

The customer outcome is a reviewable authorize service-to-service traffic at layer 4 change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.

ztunnel, Istio policy controller, Kubernetes ServiceAccounts, and destination workload; node/pod network compute; Kubernetes etcd and access telemetry; cache: not involved in policy authority.

This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.

Apply the exact L4 boundary

An imprecise example could teach an authority that the repository does not grant, so use one exact destination policy from the L4 component. The other four policies follow the same destination-owned pattern with their required callers.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: generator-l4-boundary
  namespace: helixworks-forge
spec:
  selector:
    matchLabels: { app.kubernetes.io/name: generator }
  action: ALLOW
  rules:
    - from: [{ source: { principals: [cluster.local/ns/helixworks-forge/sa/control-plane] } }]

This exact source is gitops/apps/forge/mesh/ambient-l4/authorization.yaml. At this stage generator accepts an authenticated connection from the control-plane ServiceAccount; evidence is not allowed, and no waypoint exists. The policy uses a workload selector because ztunnel enforces this L4 boundary directly. Day 33 later patches all five L4 rules to accept only the waypoint principal before adding Service-targeted L7 policy (Istio policy migration).

Policy YAML, rendered manifests, and kubectl get output are static checks. They can prove the intended principal and admission rules exist, but they cannot prove a live direct connection is denied. Apply only the L4 overlay, run its scoped verifier for object state and the allowed POST, then execute the checked-in bounded probe.

Paired allow-bounded-bypass-observation-egress and allow-bounded-bypass-observation-ingress policies admit evidence-to-generator on ports 8080 and 15008, removing NetworkPolicy denial as the explanation for that experiment. A separate broker-to-generator path has no exception and must still return the exact kindnet timeout status 42; this positive proof of CNI enforcement prevents a broken primary network policy engine from being mistaken for mesh evidence.

: "${MESH_CONTEXT:?set the exact kubectl context}"
kubectl --context "$MESH_CONTEXT" apply -k gitops/apps/forge/overlays/ambient-l4-local
MESH_CONTEXT="$MESH_CONTEXT" ./scripts/verify-istio-ambient.sh l4
SOURCE_POD="$(kubectl --context "$MESH_CONTEXT" -n helixworks-forge get pods -l app.kubernetes.io/name=evidence -o jsonpath='{.items[0].metadata.name}')"
SOURCE_POD_UID="$(kubectl --context "$MESH_CONTEXT" -n helixworks-forge get pod "$SOURCE_POD" -o jsonpath='{.metadata.uid}')"
TARGET_POD="$(kubectl --context "$MESH_CONTEXT" -n helixworks-forge get pods -l app.kubernetes.io/name=generator -o jsonpath='{.items[0].metadata.name}')"
TARGET_POD_UID="$(kubectl --context "$MESH_CONTEXT" -n helixworks-forge get pod "$TARGET_POD" -o jsonpath='{.metadata.uid}')"
NETWORK_DENY_SOURCE_POD="$(kubectl --context "$MESH_CONTEXT" -n helixworks-forge get pods -l app.kubernetes.io/name=broker -o jsonpath='{.items[0].metadata.name}')"
NETWORK_DENY_SOURCE_POD_UID="$(kubectl --context "$MESH_CONTEXT" -n helixworks-forge get pod "$NETWORK_DENY_SOURCE_POD" -o jsonpath='{.metadata.uid}')"
printf 'review source=%s/%s target=%s/%s cni-control=%s/%s\n' "$SOURCE_POD" "$SOURCE_POD_UID" "$TARGET_POD" "$TARGET_POD_UID" "$NETWORK_DENY_SOURCE_POD" "$NETWORK_DENY_SOURCE_POD_UID"
MESH_EVIDENCE_DIR="$(mktemp -d "${TMPDIR:-/tmp}/forge-mesh-l4.XXXXXX")"
export SOURCE_POD SOURCE_POD_UID TARGET_POD TARGET_POD_UID NETWORK_DENY_SOURCE_POD NETWORK_DENY_SOURCE_POD_UID MESH_EVIDENCE_DIR
MESH_L4_PROBE_APPROVED=1 ./scripts/probe-istio-l4-authorization.sh

Approval comes only after reviewing the printed names and UIDs and confirming MESH_CONTEXT is the intended local cluster. The wrapper fixes MESH_PROBE_STAGE=l4; the underlying script vetoes a waypoint label, requires generator's control-plane principal, rejects any environment other than local, and refuses ambiguous or changed workloads.

The evidence directory must contain new target.txt, network-policy.yaml, authorization-policy.yaml, denial-stderr.txt, ztunnel-observation.log, ztunnel-denial-record.log, and controls.txt files. controls.txt records successful authenticated POST /generate before and after plus the independent kindnet timeout.

For the admitted mesh experiment, the evidence Pod connects to the exact generator Pod IP, sends GET /healthz, and reads up to 4096 bytes. Any response byte is a fail-open result and aborts the run. Timeout, connection reset, broken pipe, connection abort, and zero-byte EOF are only candidate transport-denial outcomes; an unsupported exception is inconclusive. None of those candidates independently proves Istio authorization.

The probe starts a microsecond-precision UTC window, then writes ztunnel-denial-record.log only when one ztunnel log record contains the exact generator Pod IP and port, spiffe://cluster.local/ns/helixworks-forge/sa/evidence, and a policy-denial marker. Finding those fields across different records or accepting a merely nonempty log is insufficient.

Only the joined evidence supports the claim: kindnet independently denies its control path, NetworkPolicy admits the experiment, the HTTP attempt returns no response bytes with one accepted candidate status, one destination-ztunnel record correlates target, identity, and policy, all pinned UIDs stay unchanged, and both L4 POST controls pass. No GET/403 or HTTP policy claim belongs to Day 32.

Ambient enrollment previously authenticated transport without granting destination-specific connection authority. The new L4 stage requires generator's exact control-plane identity and proves that evidence is denied before any waypoint or HTTP policy exists.

L4 authorization answers who may connect, not which HTTP action they may perform.

• Use authenticated ServiceAccount principals instead of Pod IPs as workload authority.
• Keep Kubernetes NetworkPolicy, mesh identity, application tokens, and generated-app roles as separate complementary controls.
• Require independent CNI enforcement, admitted HTTP send/read, zero response bytes, an accepted candidate transport result, same-record destination-ztunnel evidence, and before/after L4 POST controls before claiming enforcement.

The boundary is credible only when identity, policy, denial, and positive control agree.

• [ ] ambient-l4-local applies five L4 policies while waypoint and L7 objects remain absent
• [ ] Reviewed evidence, generator, and broker Pod names and immutable UIDs are recorded
• [ ] Paired NetworkPolicies admit the bounded path on ports 8080 and 15008; independent kindnet control returns status 42
• [ ] Evidence-to-generator sends GET and returns zero response bytes; only timeout/reset/broken-pipe/abort/zero-byte EOF is retained as a candidate
• [ ] One sealed ztunnel record correlates target IP:port, evidence identity, and policy denial in the exact window
• [ ] Authenticated control-plane POST succeeds before and after; no GET/403 claim is made